## DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework

Fan Zhang , Naye Ji ( /a0 ) , Fuxing Gao , Bozuo Zhao , Jingmei Wu , Yanbing Jiang , Hui Du , Zhenqing Ye , Leyao Yan , Jiayang Zhu , WeiFan Zhong , Xiaomeng Ma

Abstract -Speech-driven gesture generation is an emerging domain within virtual human creation, where current methods predominantly utilize Transformer-based architectures that necessitate extensive memory and are characterized by slow inference speeds. In response to these limitations, we propose DiM-Gestures , a novel end-to-end generative model crafted to create highly personalized 3D full-body gestures solely from raw speech audio, employing Mamba-based architectures. This model integrates a Mamba-based fuzzy feature extractor with a nonautoregressive Adaptive Layer Normalization (AdaLN) Mamba2 diffusion architecture. The extractor, leveraging a Mamba framework and a WavLM pre-trained model, autonomously derives implicit, continuous fuzzy features, which are then unified into a singular latent feature. This feature is processed by the AdaLN Mamba-2, which implements a uniform conditional mechanism across all tokens to robustly model the interplay between the fuzzy features and the resultant gesture sequence. This innovative approach guarantees high fidelity in gesturespeech synchronization while maintaining the naturalness of the gestures. Employing a diffusion model for training and inference, our framework has undergone extensive subjective and objective evaluations on the ZEGGS and BEAT datasets. These assessments substantiate our model's enhanced performance relative to contemporary state-of-the-art methods, demonstrating competitive outcomes with the DiTs architecture (Persona-Gestors) while optimizing memory usage and accelerating inference speed. Code can be accessed at https://github.com/zf223669/DiMGestures.

Index Terms -Speech-driven, Gesture synthesis, Fuzzy inference, AdaLN, Diffusion, Mamba, SSMs.

## I. INTRODUCTION

R ECENT advancements have significantly broadened the scope of 3D virtual human technology, with its applications permeating diverse fields such as animation, gaming, human-computer interaction, and digital reception. Central to this research is the development of credible, personalized cospeech gestures. Speech-driven gesture generation, facilitated by deep learning, offers an efficient alternative to traditional motion capture systems, which typically require extensive manual input.

Fan Zhang, Naye Ji, Zhenqing Ye, Jiayang Zhu, WeiFan Zhong, Leyao Yan, Yanbing Jiang, Hui Du, Fuxing Gao, are with the College of Media Engineering, Communication University of Zhejiang, China; (e-mail: fanzhang@cuz.edu.cn;jinaye@cuz.edu.cn; 1767641184@qq.com; zjy190127 26077@foxmail.com; 2633252395@qq.com; yanleyao8@gmail.com; duhui@ cuz.edu.cn; fuxing@cuz.edu.cn;)

Jingmei Wu, Xiaomeng Ma, are with the School of Broadcast Announcing Arts, Communication University of Zhejiang; (e-mail: 20190095@cuz.edu.cn; 1256914539@qq.com)

Zhao Bozuo is with Changjiang Academy of Art and Design, Shantou University, China (e-mail: bzzhao@stu.edu.cn)

However, a fundamental challenge within this domain is the identification and integration of numerous input conditions essential for effective gesture synthesis. This complexity stems from a variety of factors, such as acoustic nuances, semantic content, emotional expressions, personality traits, and demographic elements like gender and age. These components collectively influence the dynamics of co-speech gestures, necessitating sophisticated models that can interpret and synthesize naturalistic human movements based on verbal communication.

Previous approaches [1]-[7] have employed manual labels and diverse feature inputs to facilitate the synthesis of personalized gestures. However, these methods rely heavily on varied unstructured feature inputs and necessitate complex multimodal processing, posing significant barriers to the practical application and broader adoption of virtual human technologies.

The fuzzy inference strategy, derived from the concept of fuzzy logic [8], has proven particularly useful in fields requiring the management of uncertain or imprecise information. This approach is noted for its effectiveness in applications such as speech-emotion recognition [9] and audio classification [10]. Unlike methods that rely on explicit classification outputs, fuzzy inference provides a spectrum of feature information, transitioning from a limited explicit discrete space to an expansive implicit continuous fuzzy space. This fuzzy space aligns more closely with real-world scenarios. Psychological research underscores the importance of various factors in speech [11]-[14], which, when considered as fuzzy features, are intricately linked to co-speech gestures. Zhang et al. [15] pioneered the successful integration of fuzzy inference strategies for generating personalized co-speech gestures.

Achieving high gesture-speech synchronization while maintaining naturalness poses a significant challenge in the field of speech-driven gesture generation. Recent advancements have pivoted towards employing Transformer and Diffusion-based models, enhancing the efficiency and flexibility of gesturegeneration technologies. Notable innovations in this domain include Diffuse Style Gesture [2], Diffuse Style Gesture+ [3], GestureDiffuClip [16], and LDA [4]. However, these methods sometimes struggle with maintaining an optimal correlation between gesture and speech, which can detract from the naturalness of the gestures produced.

The emergence of Diffusion Transformers (DiTs) in fields such as text-to-image generation [17] and text-to-video tasks like Sora 1 , which utilize Adaptive Layer Normalization (AdaLN), represents a substantial leap forward. These models introduce a uniform conditional mechanism across all tokens, significantly improving the representation of conditional and output features. This enhancement is poised to effectively model the complex relationship between speech and gestures. Despite the Persona-Gestor [15] architecture's ability to generate high-quality co-speech gestures using DiT, it faces challenges such as high memory demands and slower inference speeds due to its convolutional and transformerbased structure.

The Mamba-bases frameworks [18], [19], a relatively new entrant, has garnered attention for its simplicity, efficiency, and flexibility. These findings also underscore the Mamba architecture as a formidable competitor to the Transformer architecture, offering compelling alternatives in handling complex tasks efficiently.

In this research, we introduce DiM-Gesture , a groundbreaking model designed to synthesize personalized gestures exclusively from raw speech audio. This approach employs a fuzzy feature inference strategy within its condition extractor, utilizing the Mamba architecture, and integrates an AdaLN Mamba2 in a diffusion-based module. DiM-Gesture transitions from relying on explicit conditions to a refined, continuous representation of fuzzy features, effectively capturing a wide range of stylistic nuances and specific audio details. These features are amalgamated into a unified latent representation, facilitating the synthesis of intricate 3D full-body gestures. The implementation of AdaLN significantly bolsters the model's ability to articulate the complex relationship between speech and gestures. Leveraging a diffusion process, the model is capable of generating diverse, high-fidelity gesture outputs, showcasing its effectiveness in gesture synthesis.

For clarity, our contributions are summarized as follows:

- · Introduction of a pioneering infer Mamba-based fuzzy features strategy: This strategy allows the synthesis of a broader range of personalized gestures solely from speech audio, eliminating the need for style labels or additional inputs. The Mamba-based fuzzy feature extractor significantly enhances the system's usability capabilities.
- · Integration of the AdaLN Mamba-2 architecture within the diffusion model: This architectural choice improves the modeling of the intricate interplay between speech and gestures. Our findings validate that the Mamba architecture is a formidable competitor to traditional Transformer architectures, demonstrating an optimal balance of naturalness and synchronization in gesture generation.
- · Extensive subjective and objective evaluations: These evaluations confirm that our model surpasses current state-of-the-art approaches, highlighting the exceptional capability of our method to generate credible, speechappropriate, and personalized gestures.

## II. RELATED WORK

The discussion presented provides a concise overview of Transformer-based and diffusion-based generative models within the realm of speech-driven gesture generation.

DiffMotion [20] marks an innovative use of diffusion models in gesture synthesis, integrating an LSTM to generate diverse gestures. Alexanderson et al. [4] have refined DiffWave by replacing dilated convolutions to optimize the potential of transformer architectures. Conformers [21] employ classifierfree guidance to enhance style expression. GestureDiffuCLIP [16] utilizes transformers and AdaIN layers to integrate style guidance within the diffusion process. DiffuseStyleGesture (DSG) [2] and its extension DSG+ [3] incorporate crosslocal attention and layer normalization within their transformer models. However, these methodologies often face challenges in striking an optimal balance between gesture and speech synchronization, potentially leading to gestures that seem either underrepresented or excessively matched to the speech. Persona-Gestor [15] introduces a fuzzy feature extractor that leverages a 1D convolution to process raw speech audio for feature extraction, combined with an Adaptive Layer Normalization (AdaLN) transformer [17] to model the intricate correlation between these features and the resulting gesture sequence. While this method achieves superior motion quality, the significant memory requirements and slower inference speeds of convolutional and transformer architectures remain a challenge.

In this study, we implement a fuzzy feature inference strategy to implicitly capture nuanced features within speech audio. We synthesize natural, personalized co-speech gestures exclusively based on raw speech audio, leveraging the Mamba architecture. Furthermore, we employ the AdaLN Mamba2 architecture in place of traditional AdaLN transformers. This adaptation preserves the model's ability to capture the complex interplay between speech and gestures while ensuring the generation of high-quality actions. Notably, this approach significantly reduces memory requirements and enhances inference speed, offering a more efficient alternative for gesture synthesis in virtual human interactions.

## III. SYSTEM OVERVIEW

DiM-Gesture, an end-to-end Mamba and diffusion-based architecture, processes raw speech audio as its sole input to synthesize personalized gestures. This model adeptly balances naturalness with synchronized alignment to the corresponding speech, ensuring the gestures it generates are both realistic and timely, enhancing the interaction experience in virtual environments.

## A. Problem Formulation

In this study, we frame the problem of co-speech gesture generation as a sequence-to-sequence translation challenge, where the objective is to map a sequence of speech audio features, denoted as a = a 1: T = [ a , . . . , a 1 t , . . . , a T ] ∈ R T , to a corresponding sequence of full-body gesture features, represented as g 0 = g 0 1: T = [ g 0 1 , . . . , g 0 t , . . . , g 0 T ] ∈ R T × ( D +6) . Here, each g 0 t ∈ R ( D +6) comprises 3D joint angles alongside

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-d6a61e60-f86d-4b0b-9061-e6c5a5ecba58.jpg)

AdaLN Transformer (above)

Fig. 1: The architecture of DiM-Gesture strategically incorporates a Mamba-based fuzzy feature extractor and an Adaptive Layer Normalization (AdaLN) Mamba-2 diffusion architecture. At its core, the fuzzy feature extractor features a dual-component system engineered to capture both the nuanced style and detailed audio features. These unified latent features are subsequently channeled into the AdaLN Mamba-2. This module is pivotal in modeling the intricate relationship between the incoming audio features and the corresponding gestures. It facilitates the precise estimation of diffusion noise within the diffusion model, ensuring the generation of diverse gestures. The overall schematic includes three main components: (a) Overall Schematic, (b) Mamba-based Style Fuzzy Feature Extractor, and (c) AdaLN Mamba-2 Block.

root positional and rotational velocities at frame t , with D indicating the number of joint channels.

synthesis of personalized gestures from speech audio. These components include:

We define the probability density function (PDF), p θ ( ) · , to approximate the actual gesture data distribution p ( ) · , facilitating efficient sampling. Our aim is to generate a nonautoregressive whole pose sequence ( g 0 ) from its conditional probability distribution given the audio sequence ( a ) as a covariate:

$$g ^ { 0 } \sim p _ { \theta } ( g ^ { 0 } \, | \, a ) \approx p ( \cdot ) \coloneqq p ( g ^ { 0 } \, | \, a )$$

This formulation underscores the use of a denoising diffusion model trained to approximate the true conditional distribution of gestures given speech, illustrating the direct relationship between the audio inputs and gesture outputs, thus setting the stage for our model to learn this complex mapping effectively.

## B. Model Architecture

The architecture of DiM-Gesture, illustrated in Figure 1, is composed of four main components designed to streamline the

1) Mamba-based Style Fuzzy Feature Extractor, 2) AdaLN Mamba-2 Architecture, 3) Gesture Encoder and Decoder, and 4) Diffusion Network.

Together, these components form a cohesive system that not only captures the complexity of human gestures relative to speech but also enhances the quality and personalization of the generated gestures.

1) Mamba-based Fuzzy Feature Extractor: This module employs a fuzzy inference strategy, which means it does not rely on explicit classification outputs. Instead, it provides implicit, continuous, and fuzzy feature information, automatically learning and inferring the global style and specific details directly from raw speech audio. Illustrated in Figure 1b and Figure 2, this module is a dual-component extractor comprising both global and local extractors. The local extractor utilizes the WavLM large-scale pre-trained model [22] to process the audio sequence into discrete tokens. We selected WavLM for its proficiency in capturing the complex attributes of speech audio, which allows it to effectively represent universal audio

latent features, denoted as z a . This sophisticated approach ensures a nuanced translation of audio features into gesturerelevant data, enhancing the fidelity and personalization of the generated gestures.

We implement a global style extractor within our system, utilizing the Mamba module [18] to process z a , the universal audio latent representations. This global extractor is adept at automatically capturing and embedding global fuzzy style information from z a , producing an output token z last ∈ R 1 × D ′ , which is the last output of the Selective State Space Model(SSM). This token is subsequently broadcasted and amalgamated with z a ∈ R T ′ × D ′ to forge a unified latent representation z l ∈ R T × D ′′ . By merging both local and global insights, our architecture enhances the representational fidelity of the entire sequence, tailored for co-speech gesture generation. This unified latent representation is then channeled to the downsampling module for further refinement.

The downsampling module, crucial for aligning each latent representation with its corresponding sequence of encoded gestures, is seamlessly integrated into the condition extractor. Initially, we explored linear alignment strategies, similar to those employed in DSG [2] and DSG+ [3]. However, these methods often led to issues such as foot-skating. To address this, we have implemented a Conv1D layer with a kernel size of 201 within our architecture. This configuration enables the mapping of every 201-length target token output from the WavLM to a single gesture frame, thereby enhancing the precision of gesture synchronization. The output of this module, c 1 , serves as a unified latent representation that encapsulates both encoded audio features and the diffusion time step n , ensuring a coherent and accurate gesture generation process.

Fig. 2: An overview of the Mamba-based style fuzzy inference condition extractor.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-5017a1c1-cd94-452d-9524-655e7c302200.jpg)

2) AdaLN Mamba-2: The AdaLN's fundamental purpose is to incorporate a conditional mechanism that uniformly applies a specific function across all tokens, thereby significantly improving the model's capacity for representing both conditional and output features with enhanced efficiency. It offers a more sophisticated and nuanced approach to modeling, enabling the system to capture and articulate the complex dynamics between various input conditions and their corresponding outputs. Consequently, this leads to an improvement in the model's predictive accuracy and ability to generate outputs more aligned with the given conditions.

Diffusion Transformers (DiTs) exemplify a sophisticated advancement in diffusion model architectures, incorporating an AdaLN-infused transformer framework primarily for textto-image synthesis. This methodological enhancement has substantially lowered Fr´ echet Inception Distance (FID) scores, demonstrating improved image quality. The utility of DiTs has recently expanded to include text-conditional video generation, illustrating their versatility. Furthermore, DiTs have shown potential in co-speech gesture generation [15], marking a significant step in applying these models to sequence-based tasks. However, the inherent quadratic space complexity associated with Transformers results in substantial memory consumption and slower inference speeds, presenting practical challenges in real-time applications.

In contrast to traditional Diffusion Transformers, our approach integrates the Mamba-2 architecture [19] as a replacement for the conventional transformer module. This strategic adaptation leverages the minimal memory footprint and enhanced processing efficiency of Mamba-2, significantly accelerating inference speeds without sacrificing output quality. This novel substitution is pivotal in addressing the challenges associated with real-time, speech-driven gesture synthesis, ensuring efficient and high-quality performance.

The module involves regressing the dimension-wise scale and shift parameters ( γ and β ), derived from the fuzzy feature extractor output c 1: T , rather than directly learning these parameters, as depicted in Figure 1c. In each AdaLN Mamba-2 stack, a latent feature z n 1: T,m is generated, combining condition information and gesture using AdaLN and the Mamba-2 architecture. The index m ranges from 1 to M , where M represents the total number of AdaLN Mamba-2 stacks. Furthermore, as illustrated in Figure 1b, the final layer utilizes the same fuzzy features, supplemented by a scale and shift operation to finetune the gesture synthesis.

This method enables the creation of detailed gesture sequences directly from speech audio, obviating the need for discrete style labels or additional inputs. It significantly enhances the model's ability to generate personalized and contextually aligned gestures, providing a refined and context-sensitive gesture synthesis capability. Moreover, compared with the AdaLN transformer, the Mamba-2 architecture reduces the memory footprint and improves inference speed, while still ensuring high generation quality.

3) Gesture Encoder and Decoder: The architecture of the gesture encoder and decoder is designed to process the gesture sequence, as illustrated in Fig. 1a and Fig. 1b. The gesture encoder employs a Convolution1D layer with a kernel size of 3 to encode the initial sequence of gestures g into a hidden state h ∈ R T × D ′′ . Our experimental results indicate that a kernel size of 1 tends to produce animation jitter. In contrast, a kernel size of 3 effectively mitigates this issue by capturing the

spatial-temporal relationships inherent in gesture sequences.

The gesture decoder then reduces the feature dimension of the transformer's output D ′′ back to the original dimension D , which corresponds to the number of channels representing skeleton joints, resulting in the output of the predicted noise ( ϵ θ ). Employing a 1D kernel of size 1 in the input sequence allows the model to effectively extract meaningful features and relationships between adjacent joint channels, enhancing the quality and coherence of the generated gestures.

## C. Training and Inferencing with DDPM

The diffusion process in our architecture is crucial for reconstructing the conditional probability distribution between gestures and fuzzy features. This involves employing a systematic sampling approach from this reconstructed distribution, facilitating the generation of diverse gestures.

Following our previous work, Persona-Gestor [15], incorporating the Denoising Diffusion Probabilistic Model (DDPM) into our approach.

The model operates through two principal processes: the diffusion process and the generation process. In the diffusion process during training, the model gradually transforms the original gesture data ( g 0 ) into white noise ( g N ) by optimizing a variational bound on the data likelihood. This transformation is characterized by progressively adding noise to the data in a controlled manner.

During inference, the generation process endeavors to reverse this transformation. It recovers the original data from the noise by reversing the noising process through a Markov chain, employing Langevin sampling [23]. This technique ensures the effective and accurate reconstruction of the gesture data from its noised state. The Markov chains utilized in both the diffusion and generation processes ensure a coherent and systematic transition between stages, thereby maintaining the integrity and quality of the generated gestures. This dualprocess framework allows the model to efficiently handle and synthesize complex gesture data, reflecting the dynamic nature of human movement. The Markov chains in the diffusion process and the generation process are:

$$r \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \ \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \quad \cdots \$$

where α n := 1 -β n and α n := ∏ n i =1 α i . As shown by [24], β n is a increasing variance schedule β , ..., β 1 N with β n ∈ (0 1) , , and ˜ β n := 1 -α n -1 1 -α n β n .

The training objective centers on optimizing the parameters θ by minimizing the Negative Log-Likelihood (NLL). This is achieved via the Mean Squared Error (MSE) loss, which quantifies the discrepancy between the true noise, represented by ϵ ∼ N (0 , I ) , and the predicted noise ϵ θ :

$$\mathbb { E } _ { g _ { 1 \colon T } ^ { 0 }, \epsilon, n } [ | | \epsilon - \epsilon _ { \theta } \left ( \sqrt { \overline { \alpha } ^ { n } g ^ { 0 } } + \sqrt { 1 - \overline { \alpha } ^ { n } } \epsilon, a _ { 1 \colon T }, n \right ) | | ^ { 2 } ], \ \ ( 2 ) \quad A. \ D a t c$$

Here ϵ θ is a neural network (see figure 1a), which uses input g 0 t , a t -1 and n that to predict the ϵ , and contains the

Algorithm 1: Training for the whole sequence gesture

Input:

data

g

0

(

0

)

1:

T

∼

p

g

|

a

1:

T

and

a

1:

T

repeat

Initialize n ∼ Uniform (1 , ..., N ) and ϵ ∼ N (0 , I ) Take the gradient step on

$$\lambda _ { \theta } | | \epsilon - \epsilon _ { \theta } \left ( \sqrt { \overline { \alpha } _ { n } } g _ { 1 \colon T } ^ { 0 } + \sqrt { 1 - \overline { \alpha } _ { n } } \epsilon, a _ { 1 \colon T }, n \right ) | | ^ { 2 }$$

until converged ;

similar architecture employed in [25]. The complete training procedure is outlined in Algorithm 1.

After the training phase, we employ variational inference to generate a complete sequence of new gestures that match the original data distribution ( g 0 ∼ p θ ( g 0 | a ) ). This is facilitated by the sampling procedure detailed in Algorithm 2. During this process, we sample the entire sequence g 0 from the learned distribution.

The term σ θ represents the standard deviation of the conditional distribution p θ ( g n -1 | g n ) , which is crucial for accurately capturing the variability and intricacies of the transition between different diffusion stages. For our model, we set σ θ := ˜ β n , where ˜ β n is a predetermined scaling factor that adjusts the noise level during each diffusion step, allowing for a controlled smoothing and detail enhancement in the generation process.

Algorithm

2:

Sampling

g

0

1:

T

via annealed

Langevin dynamics

Input: noise g N 1: T ∼ N (0 , I ) and raw audio waveform

$$\text{pau}. \ \max _ { \substack { y _ { 1 \colon T } \\ a _ { 1 \colon T } } } ^ { \infty } \\ \text{for } n = N \ \text{to } 1 \ \text{do} \\ \text{if } n > 1 \ \text{then} \\ z \sim \mathcal { N } ( 0, I ) \\ \text{else}$$

$$\begin{matrix} \\ \end{matrix}$$

$$\begin{smallmatrix} \text{else} \\ z = 0 \end{smallmatrix}$$

$$\text{and} \quad \text{$z = 0$} \\ \text{time} _ { \substack { \text{end if} \\ g _ { 1 \colon T } ^ { n - 1 } = \\ \frac { 1 } { \sqrt { \alpha ^ { n } } } \left ( g _ { 1 \colon T } ^ { n } - \frac { \beta ^ { n } } { \sqrt { 1 - \overline { \alpha ^ { n } } } } \epsilon _ { \theta } \left ( g _ { 1 \colon T } ^ { n }, a _ { 1 \colon T }, n \right ) \right ) + \sqrt { \sigma _ { \theta } } z } \\ \text{end for} \\ ( 1 ) \ \frac { \text{Return: $g_{1 \colon T } ^ { 0 } } { \text{return} } }$$

During inference, we send the entire sequence of raw audio to the condition extractor component. The output of this component is then fed to the diffusion model to generate the whole sequence of accompanying gestures ( g 0 ).

## IV. EXPERIMENTS

To validate our approach, we utilized two co-speech gesture datasets: ZEGGS [7] and BEAT [26]. Our experiments focused on producing full 3D body gestures, including finger motions and locomotion.

## A. Dataset and Data Processing

1) Datasets: The ZEGGS dataset features a comprehensive collection of emotional expressions, enabling the exploration

of gesture generation across a spectrum of emotions. Conversely, the BEAT dataset is distinguished by its focus on personalized movements, capturing the unique gesture styles of various individuals. This diversity in datasets allows for robust testing and enhancement of models aimed at generating nuanced and contextually appropriate co-speech gestures.

- 2) Speech Audio Data Process: Due to the WavLM large model being pre-trained on speech audio sampled at 16 kHz, we uniformly resampled all audio from the ZEGGS and BEAT datasets down from 44.1 kHz to match this frequency, ensuring compatibility and optimal performance.
- 3) Gesture Data Process: We concentrate exclusively on full-body gestures, employing the data processing techniques detailed by Alexanderson et al. [27]. This includes capturing translational and rotational velocities to accurately delineate the root's trajectory and orientation. The datasets are uniformly downsampled to a frame rate of 20 fps. To ensure precise and continuous representation of joint angles, we utilize the exponential map technique [28]. All data are segmented into 20-second clips for training and validation purposes. For user evaluation, the generated gesture sequences are divided into 10-second clips to enhance evaluation efficiency.

## B. Model Settings

Our experiments utilize three AdaLN Mamba-2 blocks, with each Mamba-2 configured with a 256 SSM state expansion factor, a local convolution width of 4, and a block expansion factor of 2. This encoding process transforms each frame of the gesture sequence into hidden states h ∈ R 1280 . We employ the pre-trained WavLM Large model 2 for audio processing.

The diffusion model uses a quaternary variance schedule, starting from β 1 = 1 × 10 -4 to β N = 8 × 10 -2 with a linear beat schedule, and a total of N = 1000 diffusion steps. The training batch size is set to 32 per GPU.

Our model was tested on an Intel i9 processor with a 4090 GPU, in contrast to the A100 GPU used by Persona-Gestor. The training times were approximately 6 hours for ZEGGS and 22 hours for BEAT.

## C. Visualization Results

Our system excels in generating personalized gestures that are contextually aligned with speech. By leveraging the Mamba-based fuzzy inference strategy, it autonomously derives fuzzy features directly from speech audio. The results demonstrate that our AdaLN Mamba-2 framework produces gesture actions comparable to those generated by PersonaGestor, which uses convolution global fuzzy feature extractor and AdaLN transformers.

Figure 3 illustrates the visual outcomes of gestures aligned with the emotional valence conveyed by the audio. For example, the system generates gestures of joy in response to happy audio cues (see Figure 3a) and gestures of sadness for sorrowful audio (as shown in Figure 3b). Additionally, the system can infer age-related characteristics or other nuanced states from the speech audio (as depicted in 3e and 3f). These

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-15315bd0-f80d-4eda-95ce-bcb08b9c7a13.jpg)

(a) Happy

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-7dcd8235-3619-44ac-bf05-5374f1c32802.jpg)

(b) Sad

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-623ee2d0-702d-4a0e-8dc8-42c85d557f42.jpg)

(c) Angry

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-209d77ce-f7d2-46bc-8931-ca6816b9abc6.jpg)

(d) Tired

(e) old

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-00fd312b-854a-4687-9bd2-4367ddacd932.jpg)

(f) speech

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00370v1-figure-893d554b-29fb-40ea-8b10-99ed78d6c345.jpg)

Fig. 3: Samples of gestures corresponding to various emotions are presented in the subfigure. The left side illustrates the ground truth gestures, the middle section showcases gestures generated by our architecture (DiM-Gesture), and the right side presents synthesized gestures using Persona-Gestor.

visual comparisons demonstrate that DiM-Gesture can produce co-speech gestures of comparable quality to those generated by Persona-Gestor.

## D. Subjective and Objective Evaluation

In line with established practices in gesture generation research, we conducted a series of subjective and objective evaluations to assess the co-speech gestures generated by our proposed DiM-Gesture (DiM) model.

We adopted slightly varied baselines for different datasets. For the ZEGGS dataset, we employed LDA [4], and DiffuseStyleGesture (DSG) [2]. For the BEAT dataset, we utilized the same baseline models as in ZEGGS but replaced DSG with DSG+ [3] and introduced GestureDiffuCLIP (GDC) [16] as an additional baseline model. All baseline models employed are based on transformer and diffusion architectures.

In our experiments with the ZEGGS and BEAT datasets, we extended the original LDA, DSG, and DSG+ models to encompass all styles within these datasets. Efforts to adapt LDA to include finger motions encountered significant challenges, resulting in unsatisfactory gesture-generation outcomes. Consequently, we utilized gestures generated by the LDA model, excluding finger movements, for our analysis.

1) Subjective Evaluation: For comprehensive subjective evaluations, we employ three metrics: human-likeness, appropriateness, and style-appropriateness. Human-likeness assesses the naturalness and resemblance of gestures to authentic human movements, independent of speech. Appropriateness evaluates the temporal alignment of gestures with speech rhythm, intonation, and semantics, ensuring a natural flow. Style-appropriateness measures the similarity between generated gestures and their original human counterparts.

We conducted a user study employing pairwise comparisons, as recommended by [29]. In each trial, participants were presented with two 10-second video clips side by side, generated by different models, including the Ground Truth (GT), for direct comparison. Participants were instructed to select the clip they preferred based on specified evaluation criteria. Preferences were quantified on a scale from 0 to 2, with the non-selected clip in each pair receiving an inverse score (e.g., a -2 score for the non-chosen clip if the chosen one received a score of 2). A score of zero indicated a neutral preference.

Given the extensive range of styles in the ZEGGS (19) and BEAT (30) datasets, evaluating each style individually was deemed impractical. Therefore, we employed a random selection method, assigning each participant a subset of five styles from ZEGGS and six characters from BEAT for evaluation. Importantly, none of the selected audio clips were included in the training or validation sets, ensuring the integrity of the assessments.

Thirty volunteer participants-14 males and 16 females aged between 18 and 35-were recruited for this study. All participants demonstrated a high level of English proficiency, essential for accurately interpreting and responding to the tasks involved.

One-way ANOVA and posthoc Tukey, multiple comparison tests, were conducted to determine if there were statistically significant differences among the models' scores across the three evaluation aspects. The results are presented in Table I, offering detailed insights into the performance variances observed among different models regarding human-likeness, appropriateness, and style-appropriateness.

The results from the ZEGGS and BEAT datasets show that the Ground Truth (GT) achieves the highest scores ( 0 93 . ± 1 22 . for ZEGGS and 0 65 . ± 1 16 . for BEAT), exhibiting statistically significant differences ( p &lt; 0 001 . ) in humanlikeness evaluations compared to model-generated gestures. The GT features a diverse yet limited selection of gestures, each characterized by distinct traits that enhance the realism of movements. However, these specific gestures fall into the longtail distribution of the datasets, presenting substantial challenges to the learning capabilities of the models. Furthermore, the uniqueness of these gestures significantly influences the appropriateness and style-appropriateness scores.

The evaluations of the ZEGGS and BEAT datasets showed no statistically significant differences ( p &gt; 0 05 . ) between our DiM model and Persona-Gestor (PG) across all three metrics. However, PG scored slightly higher in Human-likeness on the ZEGGS dataset, whereas DiM attained marginally higher scores in Appropriateness and Style-appropriateness. In contrast, on the BEAT dataset, DiM scored slightly higher across all subjective evaluation metrics. These outcomes demonstrate that DiM is capable of generating co-speech gestures of comparable quality to those produced by PG, affirming the robustness and effectiveness of our approach in synthesizing natural and well-aligned gestures across diverse datasets.

2) Objective Evaluation: We employ three objective evaluation metrics to assess the quality and synchronization of generated gestures: Fr´ echet Gesture Distance (FGD) in both feature and raw data spaces [30], and BeatAlign [31]. Inspired by the Fr´ echet Inception Distance (FID) [32], FGD evaluates the quality of generated gestures and has shown moderate correlation with human-likeness ratings, surpassing other objective metrics [33]. BeatAlign, on the other hand, assesses gesture-audio synchrony by calculating the Chamfer Distance between audio beats and gesture beats, thus providing insights into the temporal alignment of gestures with speech rhythms.

Table I presents our results, underscoring the state-of-the-art performance of our method in objective evaluations using the Fr´ echet Gesture Distance (FGD) metrics. Our model achieves superior performance (28.16 for ZEGGS) compared to other architectures, generating gestures that closely align with the Ground Truth (GT). It also matches the BeatAlign scores (0.67 for ZEGGS) of other models, except for GestureDiffuClip (GDC) which scores slightly higher (0.69 for BEAT), highlighting its efficacy in producing co-speech gestures that synchronize accurately with speech rhythms. Despite GDC's high BeatAlign score, corroborating user feedback indicates its overemphasis on prosodic cues results in frequent highfrequency gestures that, while technically accurate, reduce the naturalness of the gestures. On the BEAT dataset, our model's FGD scores (276.32) are marginally lower than Persona-Gestor (276.25), reflecting competitive performance in gesture quality.

TABLE I: The subject mean perceptual rating score. Bold fonts were utilized to emphasize the best results for each metric among the different methods, except for the GT.

|         | Methods                     | Subject Evaluation Metric   | Subject Evaluation Metric   | Subject Evaluation Metric   | Objective Evaluation Metric   | Objective Evaluation Metric   | Objective Evaluation Metric   |
|---------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|-------------------------------|-------------------------------|-------------------------------|
| Dataset | Model                       | Human ↑ likeness            | Appropriateness ↑           | Style ↑ appropriateness     | FGD ↓ on feature space        | FGD ↓ on raw data space       | BeatAlign ↑                   |
|         | GT                          | 0.93±1.22                   | 1.21±1.25                   | /                           | /                             | /                             | /                             |
|         | LDA                         | -0.76±1.28                  | 0.013±1.35                  | 0.38±1.87                   | 124.55                        | 50996.33                      | 0.66                          |
|         | DSG                         | -0.42±1.37                  | -0.76±1.08                  | -0.79±1.18                  | 66.77                         | 33297.50                      | 0.63                          |
| ZEGGS   | PG                          | 0.47±1.17                   | 0.476±1.29                  | 0.76±1.52                   | 28.17                         | 26193.92                      | 0.67                          |
|         | (Proposed) DiM-AdaLN-Mamba2 | 0.46±1.07                   | 0.478±1.32                  | 0.77±1.35                   | 28.16                         | 26013.12                      | 0.67                          |
|         | DiM-AdaLN-Mamba1            | 0.21±1.23                   | 0.416±1.18                  | 0.63±1.53                   | 35.21                         | 30453.23                      | 0.66                          |
|         | DiM-ConvSE                  | 0.45±1.27                   | 0.477±1.14                  | 0.77±1.27                   | 28.53                         | 26878.34                      | 0.67                          |
|         | GT                          | 0.65±1.16                   | 0.96±1.04                   | /                           | /                             | /                             | /                             |
|         | LDA                         | -1.65±0.73                  | -1.59±0.74                  | -1.35±1.05                  | 264.06                        | 3471.26                       | 0.66                          |
|         | DSG                         | -0.28±1.17                  | -0.49±1.15                  | -0.40±1.24                  | 23811.46                      | 2384465.64                    | 0.43                          |
|         | GDC                         | 0.54±1.12                   | 0.47±1.25                   | 0.30±1.27                   | 432.15                        | 93215.56                      | 0.69                          |
| BEAT    | PG                          | 0.56±1.26                   | 0.64±1.07                   | 0.66±1.33                   | 276.25                        | 3584.95                       | 0.68                          |
|         | (Proposed) DiM-AdaLN-Mamba2 | 0.57±1.04                   | 0.65±1.15                   | 0.67±1.28                   | 276.32                        | 3607.75                       | 0.68                          |
|         | DiM-AdaLN-Mamba1            | 0.37±1.15                   | 0.38±1.06                   | 0.60±1.05                   | 283.24                        | 3924.58                       | 0.66                          |
|         | DiM-ConvSE                  | 0.56±1.22                   | 0.65±1.32                   | 0.67±1.38                   | 276.58                        | 3683.65                       | 0.68                          |

## E. Ablation Studies

Ablation studies were conducted to assess the influence of crucial components within our model, particularly focusing on the Mamba-based fuzzy feature extractor and the AdaLN Mamba-2 architecture. These studies aimed to elucidate how each component contributes to the overall performance and effectiveness of the gesture synthesis process.

1) Ablation of Mamba-based Style Fuzzy Feature Extractor: For the Mamba-based style fuzzy feature extractor, we conducted an investigation into the implications of replacing the Mamba architecture with a convolutional style extractor (similar to the approach in PG), which we termed DiMConvSE. This experiment aimed to assess the differential impacts of the Mamba and convolutional methodologies on the effectiveness and fidelity of the synthesized gestures.

2) Ablation of AdaLN Mamba-2: In our ablation study, we explored the impact of different versions of the Mamba architecture by replacing the AdaLN Mamba-2 with Mamba1 [18]. This experiment was designed to assess the efficacy of the AdaLN Mamba-2 in handling the intricate dynamics of gesture generation compared to its predecessor, Mamba-1. The main focus was to determine whether the advancements in Mamba-2, particularly the integration of adaptive layer normalization (AdaLN), provide substantial improvements in gesture synthesis quality, efficiency, and model responsiveness. The results of this comparison are detailed in Table I and Table II, illustrating differences in gesture quality, computational efficiency, and performance metrics between the two Mamba versions.

The results, as detailed in Table I, reveal no statistically significant differences ( p &gt; 0 05 . ) in the style-appropriateness metrics between DiM and DiM-ConvSE for both the ZEGGS and BEAT dataset experiments. These findings underscore the equivalence of the Mamba-based style fuzzy feature extractor in performance with the convolution-based extractor, highlighting its efficacy in gesture synthesis. Additionally, as indicated in Table II, the Mamba architecture demonstrates less parameter count and lower memory consumption than DiMConvSE.

TABLE II: In the ablation study focusing on the Mamba-based style fuzzy feature extractor and the AdaLN Mamba2, we meticulously assessed the parameter count and inference time to quantify the efficiency and performance impact of these components.

|                             | Param. Count   | Inference Time (length of 80s Gesture Sequence)   |
|-----------------------------|----------------|---------------------------------------------------|
| PG                          | 1.2B           | 82s                                               |
| (Proposed) DiM-AdaLN-Mamba2 | 421M           | 30s                                               |
| DiM-AdaLN-Mamba1            | 426M           | 15s                                               |
| DiM-ConvSE                  | 711M           | 9s                                                |

In our ablation studies on the AdaLN Mamba-2, substituting the Mamba-2 module with Mamba-1 led to a significant decline in performance across all metrics. This deterioration can be attributed to the earlier architecture's inability to precisely synchronize speech rhythm and capture stylistic nuances like its successor. These findings highlight the integrated enhancements in Mamba-2, particularly its advanced capability to effectively handle the intricate dynamics of speech-driven gesture generation. However, it is worth noting that DiMAdaLN-Mamba-1 exhibits a faster inference time.

## V. DISSCUSTION AND CONCLUSION

In this study, we present DiM-Gesture , an innovative network architecture designed to generate personality-specific gestures directly from raw speech audio using a Mamba-based architecture. DiM-Gesture incorporates a Mamba-based style fuzzy feature extractor and an AdaLN Mamba-2 diffusion architecture, facilitating the seamless synthesis of nuanced, personality-driven gestures. Our approach achieves a quality of action generation comparable to that of the AdaLN Transformer architecture (Persona-Gestor) while requiring significantly less memory and substantially enhancing generation speed than Persona-Gestor. This demonstrates DiM-Gesture 's

capability to provide efficient and high-quality gesture synthesis.

The Mamba-based fuzzy feature extractor employs a fuzzy feature inference strategy within its dual-component module to autonomously infer both fuzzy stylistic features and specific audio details. These elements are merged into a unified latent representation, enabling the generation of speaker-aware personalized 3D full-body gestures. This approach integrates a pivotal innovation in synthesizing personality-driven gestures by leveraging automatically inferred fuzzy features, thereby eliminating the need for explicit style labels or additional features. Such advancements facilitate an end-to-end gesture generation process that authentically reflects the speaker's unique characteristics directly from raw speech audio. The integration of fuzzy feature inference streamlines the creation process, enhancing both generalization capabilities and user accessibility.

The AdaLN Mamba-based mechanism, a conditional architecture, uniformly applies a specific function across all sequence tokens, significantly enhancing the model's ability to capture and represent both conditional dependencies and output characteristics efficiently. Like AdaLN transformers, AdaLN Mamba-2 enhances the understanding and processing of the complex interactions between continuous fuzzy features as conditional inputs and resultant gesture synthesis. This leads to improved model performance and output fidelity. Ultimately, DiM-Gesture employs a diffusion mechanism to produce a diverse spectrum of gesture outputs, showcasing its capability to handle varied and nuanced gesture synthesis effectively.

Our study highlights key areas for enhancement: There remains a noticeable disparity between the current batch generation form and the desired real-time generation capability. Bridging this gap is crucial for applications requiring immediate gesture synthesis, such as live interactions or performances.

## VI. ACKNOWLEDGMENTS

This work was partially supported by the 'Pioneer' and 'Leading Goose' R&amp;D Program of Zhejiang (No.2023 C01212), the National Key Research and Development Program of China (No.2022YFF 0902305), the Public Welfare Technology Application Research Project of Zhejiang (No.LGF21F020002, No.LGF22F020008), the Key Program and development projects of Zhejiang Province of China (No.2021C03137), and the Key Lab of Film and TV Media Technology of Zhejiang Province (No.2020E10015).

## REFERENCES

- [1] U. Bhattacharya, E. Childs, N. Rewkowski, and D. Manocha, 'Speech2affectivegestures: Synthesizing co-speech gestures with generative adversarial affective expression learning,' in Proc. of the 29th ACM International Conf. on Multimedia , 2021, pp. 2027-2036.
- [2] S. Yang, Z. Wu, M. Li, Z. Zhang, L. Hao, W. Bao, M. Cheng, and L. Xiao, 'DiffuseStyleGesture: Stylized audio-driven co-speech gesture generation with diffusion models,' arXiv preprint arXiv:2305.04919 , 2023.
- [3] S. Yang, H. Xue, Z. Zhang, M. Li, Z. Wu, X. Wu, S. Xu, and Z. Dai, 'The diffusestylegesture+ entry to the genea challenge 2023,' arXiv preprint arXiv:2308.13879 , 2023.
- [4] S. Alexanderson, R. Nagy, J. Beskow, and G. E. Henter, 'Listen, denoise, action! audio-driven motion synthesis with diffusion models,' ACM Transactions on Graphics (TOG) , vol. 42, no. 4, pp. 1-20, 2023.
- [5] S. Yang, Z. Wang, Z. Wu, M. Li, Z. Zhang, Q. Huang, L. Hao, S. Xu, X. Wu, and C. Yang, 'Unifiedgesture: A unified gesture synthesis model for multiple skeletons,' in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 1033-1044.
- [6] J. Li, D. Kang, W. Pei, X. Zhe, Y. Zhang, L. Bao, and Z. He, 'Audio2gestures: Generating diverse gestures from audio,' IEEE Transactions on Visualization and Computer Graphics , pp. 1-15, 2023.
- [7] S. Ghorbani, Y. Ferstl, D. Holden, N. F. Troje, and M.-A. Carbonneau, 'Zeroeggs: Zero-shot example-based gesture generation from speech,' in Computer Graphics Forum , vol. 42, no. 1. Wiley Online Library, 2023, pp. 206-216.
- [8] L. A. Zadeh, 'Fuzzy sets,' Information and control , vol. 8, no. 3, pp. 338-353, 1965.
- [9] S. Vashishtha and S. Susan, 'Unsupervised fuzzy inference system for speech emotion recognition using audio and text cues (workshop paper),' in 2020 IEEE sixth international conference on multimedia big data (BigMM) . IEEE, 2020, pp. 394-403.
- [10] N. M. Patil and M. U. Nemade, 'Content-based audio classification and retrieval using segmentation, feature extraction and neural network approach,' in Advances in computer communication and computational sciences: Proceedings of IC4S 2018 . Springer, 2019, pp. 263-281.
- [11] R. A. Calvo, S. D'Mello, J. M. Gratch, and A. Kappas, The Oxford handbook of affective computing . Oxford Library of Psychology, 2015.
- [12] M. Goudbeek and K. Scherer, 'Beyond arousal: Valence and potency/control cues in the vocal expression of emotion,' The Journal of the Acoustical Society of America , vol. 128, no. 3, pp. 1322-1336, 2010.
- [13] J. Hirschberg and C. D. Manning, 'Advances in natural language processing,' Science , vol. 349, no. 6245, pp. 261-266, 2015.
- [14] K. Campbell-Kibler, 'Intersecting variables and perceived sexual orientation in men,' American Speech , vol. 86, no. 1, pp. 52-68, 2011.
- [15] F. Zhang, Z. Wang, X. Lyu, S. Zhao, M. Li, W. Geng, N. Ji, H. Du, F. Gao, and H. Wu, 'Speech-driven personalized gesture synthetics: Harnessing automatic fuzzy feature inference,' IEEE Transactions on Visualization and Computer Graphics , 2024, publisher: IEEE. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10508094/
- [16] T. Ao, Z. Zhang, and L. Liu, 'Gesturediffuclip: Gesture diffusion model with clip latents,' arXiv preprint arXiv:2303.14613 , 2023.
- [17] W. Peebles and S. Xie, 'Scalable diffusion models with transformers,' arXiv preprint arXiv:2212.09748 , 2022.
- [18] A. Gu and T. Dao, 'Mamba: Linear-time sequence modeling with selective state spaces.' [Online]. Available: http://arxiv.org/abs/2312. 00752
- [19] T. Dao and A. Gu, 'Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,' arXiv preprint arXiv:2405.21060 , 2024. [Online]. Available: https: //arxiv.org/abs/2405.21060
- [20] F. Zhang, N. Ji, F. Gao, and Y. Li, 'Diffmotion: Speech-driven gesture synthesis using denoising diffusion model,' in MultiMedia Modeling: 29th International Conf., MMM 2023, Bergen, Norway, January 9-12, 2023, Proc., Part I . Springer, 2023, pp. 231-242.
- [21] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, 'Conformer: Convolutionaugmented transformer for speech recognition,' in INTERSPEECH 2020 , ser. Interspeech, 2020, pp. 5036-5040, interspeech Conf., Shanghai, PEOPLES R CHINA, OCT 25-29, 2020.
- [22] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, and X. Xiao, 'Wavlm: Large-scale self-supervised pretraining for full stack speech processing,' IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505-1518, 2022.
- [23] L. Paul, 'sur la th´ eorie du mouvement brownien,' C. R. Acad. Sci. , vol. 65, no. 11, pp. 146,530-533, 1908, publisher: American Association of Physics Teachers.
- [24] J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Advances in Neural Information Processing Systems , vol. 33, pp. 68406851, 2020.
- [25] K. Rasul, C. Seward, I. Schuster, and R. Vollgraf, 'Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting,' in International Conf. on Machine Learning , 2021, pp. 8857-8868.
- [26] H. Liu, Z. Zhu, N. Iwamoto, Y. Peng, Z. Li, Y. Zhou, E. Bozkurt, and B. Zheng, 'Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis,' in COMPUTER VISION, ECCV 2022, PT VII , vol. 13667, 2022, pp. 612-630.

- [27] S. Alexanderson, G. E. Henter, T. Kucherenko, and J. Beskow, 'Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows,' in Computer Graphics Forum , vol. 39. Wiley Online Library, 2020, pp. 487-496, issue: 2.
- [28] F. S. Grassia, 'Practical parameterization of rotations using the exponential map,' Journal of graphics tools , vol. 3, no. 3, pp. 29-48, 1998. [29] P. Wolfert, J. M. Girard, T. Kucherenko, and T. Belpaeme, 'To rate or not to rate: Investigating evaluation methods for generated co-speech gestures,' in Proceedings of the 2021 International Conference on Multimodal Interaction . ACM, 2021-10-18, pp. 494-502.
- [30] Y. Yoon, B. Cha, J.-H. Lee, M. Jang, J. Lee, J. Kim, and G. Lee, 'Speech gesture generation from the trimodal context of text, audio, and speaker identity,' ACM Transactions on Graphics (TOG) , vol. 39, no. 6, pp. 1-16, 2020.
- [31] R. Li, S. Yang, D. A. Ross, and A. Kanazawa, 'Ai choreographer: Music conditioned 3d dance generation with aist++,' in Proc. of the IEEE/CVF International Conf. on Computer Vision , 2021, pp. 13 401-13 412.
- [32] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, 'Gans trained by a two time-scale update rule converge to a local nash equilibrium,' Advances in neural information processing systems , vol. 30, 2017.
- [33] T. Kucherenko, P. Wolfert, Y. Yoon, C. Viegas, T. Nikolov, M. Tsakov, and G. E. Henter, 'Evaluating gesture-generation in a large-scale open challenge: The genea challenge 2022,' arXiv preprint arXiv:2303.08737 , 2023.
Fan Zhang Naye Ji Fuxing Gao Bozuo Zhao Jingmei Wu Yanbing Jiang Hui Du Zhenqing Ye Jiayang Zhu WeiFan Zhong Leyao Yan Xiaomeng Ma
DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework
Speech-driven gesture generation is an emerging domain within virtual human
creation, where current methods predominantly utilize Transformer-based
architectures that necessitate extensive memory and are characterized by slow
inference speeds. In response to these limitations, we propose
\textit{DiM-Gestures}, a novel end-to-end generative model crafted to create
highly personalized 3D full-body gestures solely from raw speech audio,
employing Mamba-based architectures. This model integrates a Mamba-based fuzzy
feature extractor with a non-autoregressive Adaptive Layer Normalization
(AdaLN) Mamba-2 diffusion architecture. The extractor, leveraging a Mamba
framework and a WavLM pre-trained model, autonomously derives implicit,
continuous fuzzy features, which are then unified into a singular latent
feature. This feature is processed by the AdaLN Mamba-2, which implements a
uniform conditional mechanism across all tokens to robustly model the interplay
between the fuzzy features and the resultant gesture sequence. This innovative
approach guarantees high fidelity in gesture-speech synchronization while
maintaining the naturalness of the gestures. Employing a diffusion model for
training and inference, our framework has undergone extensive subjective and
objective evaluations on the ZEGGS and BEAT datasets. These assessments
substantiate our model's enhanced performance relative to contemporary
state-of-the-art methods, demonstrating competitive outcomes with the DiTs
architecture (Persona-Gestors) while optimizing memory usage and accelerating
inference speed.