Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation
<|reference_start|>Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation: Current co-speech motion generation approaches usually focus on upper body gestures following speech contents only, while lacking supporting the elaborate control of synergistic full-body motion based on text prompts, such as talking while walking. The major challenges lie in 1) the existing speech-to-motion datasets only involve highly limited full-body motions, making a wide range of common human activities out of training distribution; 2) these datasets also lack annotated user prompts. To address these challenges, we propose SynTalker, which utilizes the off-the-shelf text-to-motion dataset as an auxiliary for supplementing the missing full-body motion and prompts. The core technical contributions are two-fold. One is the multi-stage training process which obtains an aligned embedding space of motion, speech, and prompts despite the significant distributional mismatch in motion between speech-to-motion and text-to-motion datasets. Another is the diffusion-based conditional inference process, which utilizes the separate-then-combine strategy to realize fine-grained control of local body parts. Extensive experiments are conducted to verify that our approach supports precise and flexible control of synergistic full-body motion generation based on both speeches and user prompts, which is beyond the ability of existing approaches.<|reference_end|>
@article{chen2024enabling,
    title={Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion
  Generation},
    author={Bohong Chen, Yumeng Li, Yao-Xiang Ding, Tianjia Shao, Kun Zhou},
    journal={arXiv preprint arXiv:2410.00464},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00464},
    primaryClass={cs.CV}
}