Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation
<|reference_start|>Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation: Audio-driven talking head generation is a significant and challenging task applicable to various fields such as virtual avatars, film production, and online conferences. However, the existing GAN-based models emphasize generating well-synchronized lip shapes but overlook the visual quality of generated frames, while diffusion-based models prioritize generating high-quality frames but neglect lip shape matching, resulting in jittery mouth movements. To address the aforementioned problems, we introduce a two-stage diffusion-based model. The first stage involves generating synchronized facial landmarks based on the given speech. In the second stage, these generated landmarks serve as a condition in the denoising process, aiming to optimize mouth jitter issues and generate high-fidelity, well-synchronized, and temporally coherent talking head videos. Extensive experiments demonstrate that our model yields the best performance.<|reference_end|>
@article{tan2024landmark-guided,
    title={Landmark-guided Diffusion Model for High-fidelity and Temporally
  Coherent Talking Head Generation},
    author={Jintao Tan, Xize Cheng, Lingyu Xiong, Lei Zhu, Xiandong Li, Xianjia
  Wu, Kai Gong, Minglei Li, Yi Cai},
    journal={arXiv preprint arXiv:2408.01732},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.01732},
    primaryClass={cs.CV cs.AI}
}