StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads
<|reference_start|>StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads: Individuals have unique facial expression and head pose styles that reflect their personalized speaking styles. Existing one-shot talking head methods cannot capture such personalized characteristics and therefore fail to produce diverse speaking styles in the final videos. To address this challenge, we propose a one-shot style-controllable talking face generation method that can obtain speaking styles from reference speaking videos and drive the one-shot portrait to speak with the reference speaking styles and another piece of audio. Our method aims to synthesize the style-controllable coefficients of a 3D Morphable Model (3DMM), including facial expressions and head movements, in a unified framework. Specifically, the proposed framework first leverages a style encoder to extract the desired speaking styles from the reference videos and transform them into style codes. Then, the framework uses a style-aware decoder to synthesize the coefficients of 3DMM from the audio input and style codes. During decoding, our framework adopts a two-branch architecture, which generates the stylized facial expression coefficients and stylized head movement coefficients, respectively. After obtaining the coefficients of 3DMM, an image renderer renders the expression coefficients into a specific person's talking-head video. Extensive experiments demonstrate that our method generates visually authentic talking head videos with diverse speaking styles from only one portrait image and an audio clip.<|reference_end|>
@article{wang2024styletalk++:,
    title={StyleTalk++: A Unified Framework for Controlling the Speaking Styles of
  Talking Heads},
    author={Suzhen Wang, Yifeng Ma, Yu Ding, Zhipeng Hu, Changjie Fan, Tangjie Lv,
  Zhidong Deng, Xin Yu},
    journal={arXiv preprint arXiv:2409.09292},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09292},
    primaryClass={cs.CV}
}