title: DreamTalk: When Expressive Talking Head Generation Meets Diffusion
    Probabilistic Models
summary: Diffusion models have shown remarkable success in a variety of downstream
  generative tasks, yet remain under-explored in the important and challenging
  expressive talking head generation. In this work, we propose a DreamTalk
  framework to fulfill this gap, which employs meticulous design to unlock the
  potential of diffusion models in generating expressive talking heads.
  Specifically, DreamTalk consists of three crucial components: a denoising
  network, a style-aware lip expert, and a style predictor. The diffusion-based
  denoising network is able to consistently synthesize high-quality audio-driven
  face motions across diverse expressions. To enhance the expressiveness and
  accuracy of lip motions, we introduce a style-aware lip expert that can guide
  lip-sync while being mindful of the speaking styles. To eliminate the need for
  expression reference video or text, an extra diffusion-based style predictor is
  utilized to predict the target expression directly from the audio. By this
  means, DreamTalk can harness powerful diffusion models to generate expressive
  faces effectively and reduce the reliance on expensive style references.
  Experimental results demonstrate that DreamTalk is capable of generating
  photo-realistic talking faces with diverse speaking styles and achieving
  accurate lip motions, surpassing existing state-of-the-art counterparts.
paper_authors._id: 657fd6bb1e2087032477f5b8
paper_authors.name: Yifeng Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:31:52.194Z
paper_authors._id: 651e91c59dcfa4c1bfa6f415
paper_authors.avatarUrl: /avatars/76ee4bc1d9910bd5a23dc27d2f49d4b0.svg
paper_authors.fullname: Yifeng Ma
paper_authors.type: user
paper_authors.user: lostork
paper_authors._id: 657fd6bb1e2087032477f5b9
paper_authors.name: Shiwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:31:59.108Z
paper_authors._id: 63a1200045edac9f7508bae9
paper_authors.avatarUrl: /avatars/e84f0b045f32c5b8b4da43458650b925.svg
paper_authors.fullname: Shiwei Zhang
paper_authors.type: user
paper_authors.user: StevenZhang
paper_authors._id: 657fd6bb1e2087032477f5ba
paper_authors.name: Jiayu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:32:18.425Z
paper_authors._id: 64ec8ce1a91f5a573e4ab454
paper_authors.avatarUrl: /avatars/1c855b26abe040a4d19ade2be29fc1c2.svg
paper_authors.fullname: Jiayu Wang
paper_authors.type: user
paper_authors.user: ArseneRean
paper_authors._id: 657fd6bb1e2087032477f5bb
paper_authors.name: Xiang Wang
paper_authors._id: 657fd6bb1e2087032477f5bc
paper_authors.name: Yingya Zhang
paper_authors._id: 657fd6bb1e2087032477f5bd
paper_authors.name: Zhidong Deng
paper_title: DreamTalk: When Expressive Talking Head Generation Meets Diffusion
    Probabilistic Models
paper_summary: Diffusion models have shown remarkable success in a variety of downstream
  generative tasks, yet remain under-explored in the important and challenging
  expressive talking head generation. In this work, we propose a DreamTalk
  framework to fulfill this gap, which employs meticulous design to unlock the
  potential of diffusion models in generating expressive talking heads.
  Specifically, DreamTalk consists of three crucial components: a denoising
  network, a style-aware lip expert, and a style predictor. The diffusion-based
  denoising network is able to consistently synthesize high-quality audio-driven
  face motions across diverse expressions. To enhance the expressiveness and
  accuracy of lip motions, we introduce a style-aware lip expert that can guide
  lip-sync while being mindful of the speaking styles. To eliminate the need for
  expression reference video or text, an extra diffusion-based style predictor is
  utilized to predict the target expression directly from the audio. By this
  means, DreamTalk can harness powerful diffusion models to generate expressive
  faces effectively and reduce the reliance on expensive style references.
  Experimental results demonstrate that DreamTalk is capable of generating
  photo-realistic talking faces with diverse speaking styles and achieving
  accurate lip motions, surpassing existing state-of-the-art counterparts.
paper_ai_keywords: diffusion denoising network style-aware lip expert diffusion-based style predictor audio-driven face motions lip-sync speaking styles photo-realistic talking faces