Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
<|reference_start|>Faces that Speak: Jointly Synthesising Talking Face and Speech from Text: The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities.<|reference_end|>
@article{jang2024faces,
    title={Faces that Speak: Jointly Synthesising Talking Face and Speech from Text},
    author={Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hong-Sun Yang,
  Yoon-Cheol Ju, Il-Hwan Kim, Byeong-Yeol Kim, Joon Son Chung},
    journal={arXiv preprint arXiv:2405.10272},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10272},
    primaryClass={cs.CV cs.AI cs.SD eess.AS eess.IV}
}