DreamTalk: When Expressive Talking Head Generation Meets Diffusion
  Probabilistic Models
Diffusion models have shown remarkable success in a variety of downstream
generative tasks, yet remain under-explored in the important and challenging
expressive talking head generation. In this work, we propose a DreamTalk
framework to fulfill this gap, which employs meticulous design to unlock the
potential of diffusion models in generating expressive talking heads.
Specifically, DreamTalk consists of three crucial components: a denoising
network, a style-aware lip expert, and a style predictor. The diffusion-based
denoising network is able to consistently synthesize high-quality audio-driven
face motions across diverse expressions. To enhance the expressiveness and
accuracy of lip motions, we introduce a style-aware lip expert that can guide
lip-sync while being mindful of the speaking styles. To eliminate the need for
expression reference video or text, an extra diffusion-based style predictor is
utilized to predict the target expression directly from the audio. By this
means, DreamTalk can harness powerful diffusion models to generate expressive
faces effectively and reduce the reliance on expensive style references.
Experimental results demonstrate that DreamTalk is capable of generating
photo-realistic talking faces with diverse speaking styles and achieving
accurate lip motions, surpassing existing state-of-the-art counterparts.
657fd6bb1e2087032477f5b8
Yifeng Ma
admin_assigned
2023-12-18T09:31:52.194Z
651e91c59dcfa4c1bfa6f415
/avatars/bde9b7e833d91853ceca0bef0b91cccb.svg
Yifeng Ma
user
lostork
657fd6bb1e2087032477f5b9
Shiwei Zhang
admin_assigned
2023-12-18T09:31:59.108Z
63a1200045edac9f7508bae9
/avatars/e84f0b045f32c5b8b4da43458650b925.svg
Shiwei Zhang
user
StevenZhang
657fd6bb1e2087032477f5ba
Jiayu Wang
admin_assigned
2023-12-18T09:32:18.425Z
64ec8ce1a91f5a573e4ab454
/avatars/1c855b26abe040a4d19ade2be29fc1c2.svg
Jiayu Wang
user
ArseneRean
657fd6bb1e2087032477f5bb
Xiang Wang
657fd6bb1e2087032477f5bc
Yingya Zhang
657fd6bb1e2087032477f5bd
Zhidong Deng
DreamTalk: When Expressive Talking Head Generation Meets Diffusion
  Probabilistic Models
Diffusion models have shown remarkable success in a variety of downstream
generative tasks, yet remain under-explored in the important and challenging
expressive talking head generation. In this work, we propose a DreamTalk
framework to fulfill this gap, which employs meticulous design to unlock the
potential of diffusion models in generating expressive talking heads.
Specifically, DreamTalk consists of three crucial components: a denoising
network, a style-aware lip expert, and a style predictor. The diffusion-based
denoising network is able to consistently synthesize high-quality audio-driven
face motions across diverse expressions. To enhance the expressiveness and
accuracy of lip motions, we introduce a style-aware lip expert that can guide
lip-sync while being mindful of the speaking styles. To eliminate the need for
expression reference video or text, an extra diffusion-based style predictor is
utilized to predict the target expression directly from the audio. By this
means, DreamTalk can harness powerful diffusion models to generate expressive
faces effectively and reduce the reliance on expensive style references.
Experimental results demonstrate that DreamTalk is capable of generating
photo-realistic talking faces with diverse speaking styles and achieving
accurate lip motions, surpassing existing state-of-the-art counterparts.
DreamTalk enables expressive talking head generation using diffusion models with specialized components for denoising, lip-sync, and style prediction.
denoising network style-aware lip expert style predictor diffusion models audio-driven face motions lip-sync target expression prediction photo-realistic talking faces diverse speaking styles accurate lip motions