title: DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for
    Single Image Talking Face Generation
summary: The generation of emotional talking faces from a single portrait image
  remains a significant challenge. The simultaneous achievement of expressive
  emotional talking and accurate lip-sync is particularly difficult, as
  expressiveness is often compromised for the accuracy of lip-sync. As widely
  adopted by many prior works, the LSTM network often fails to capture the
  subtleties and variations of emotional expressions. To address these
  challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
  framework, tailored for generating diverse expressions and accurate lip-sync
  concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
  that generates diverse highly dynamic emotional expressions and head poses in
  accordance with the audio and the referenced emotion style. Given the strong
  correlation between lip motion and audio, we then refine the dynamics with
  enhanced lip-sync accuracy using audio features and emotion style. To this end,
  we deploy a video-to-video rendering module to transfer the expressions and lip
  motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
  and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
  expressiveness, lip-sync accuracy and perceptual quality.
paper_authors._id: 6584edf9e07563d421f639c2
paper_authors.name: Chenxu Zhang
paper_authors._id: 6584edf9e07563d421f639c3
paper_authors.name: Chao Wang
paper_authors._id: 6584edf9e07563d421f639c4
paper_authors.name: Jianfeng Zhang
paper_authors._id: 6584edf9e07563d421f639c5
paper_authors.name: Hongyi Xu
paper_authors._id: 6584edf9e07563d421f639c6
paper_authors.name: Guoxian Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T11:05:22.392Z
paper_authors._id: 63086a237dc1b1a54cc6c24d
paper_authors.avatarUrl: /avatars/477b94134edc4c18c8f769ecbb7d8091.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: guoxiansong
paper_authors._id: 6584edf9e07563d421f639c7
paper_authors.name: You Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T11:05:42.961Z
paper_authors._id: 6408dfd4b6a334f53e24023c
paper_authors.avatarUrl: /avatars/b7e3fa4fbec6313e94ff3384b74dabfc.svg
paper_authors.fullname: You Xie
paper_authors.type: user
paper_authors.user: youxie
paper_authors._id: 6584edf9e07563d421f639c8
paper_authors.name: Linjie Luo
paper_authors._id: 6584edf9e07563d421f639c9
paper_authors.name: Yapeng Tian
paper_authors._id: 6584edf9e07563d421f639ca
paper_authors.name: Xiaohu Guo
paper_authors._id: 6584edf9e07563d421f639cb
paper_authors.name: Jiashi Feng
paper_title: DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for
    Single Image Talking Face Generation
paper_summary: The generation of emotional talking faces from a single portrait image
  remains a significant challenge. The simultaneous achievement of expressive
  emotional talking and accurate lip-sync is particularly difficult, as
  expressiveness is often compromised for the accuracy of lip-sync. As widely
  adopted by many prior works, the LSTM network often fails to capture the
  subtleties and variations of emotional expressions. To address these
  challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
  framework, tailored for generating diverse expressions and accurate lip-sync
  concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
  that generates diverse highly dynamic emotional expressions and head poses in
  accordance with the audio and the referenced emotion style. Given the strong
  correlation between lip motion and audio, we then refine the dynamics with
  enhanced lip-sync accuracy using audio features and emotion style. To this end,
  we deploy a video-to-video rendering module to transfer the expressions and lip
  motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
  and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
  expressiveness, lip-sync accuracy and perceptual quality.
paper_ai_summary: DREAM-Talk is a two-stage diffusion-based framework generating expressive emotional talking faces with accurate lip-sync from single portrait images.
paper_ai_keywords: LSTM network diffusion-based audio-driven framework EmoDiff diffusion module audio features video-to-video rendering module proxy 3D avatar