DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework
  for Talking Head Video Generation
Talking head generation intends to produce vivid and realistic talking head
videos from a single portrait and speech audio clip. Although significant
progress has been made in diffusion-based talking head generation, almost all
methods rely on autoregressive strategies, which suffer from limited context
utilization beyond the current generation step, error accumulation, and slower
generation speed. To address these challenges, we present DAWN (Dynamic frame
Avatar With Non-autoregressive diffusion), a framework that enables all-at-once
generation of dynamic-length video sequences. Specifically, it consists of two
main components: (1) audio-driven holistic facial dynamics generation in the
latent motion space, and (2) audio-driven head pose and blink generation.
Extensive experiments demonstrate that our method generates authentic and vivid
videos with precise lip motions, and natural pose/blink movements.
Additionally, with a high generation speed, DAWN possesses strong extrapolation
capabilities, ensuring the stable production of high-quality long videos. These
results highlight the considerable promise and potential impact of DAWN in the
field of talking head video generation. Furthermore, we hope that DAWN sparks
further exploration of non-autoregressive approaches in diffusion models. Our
code will be publicly at https://github.com/Hanbo-Cheng/DAWN-pytorch.
67147d177b711cff63c0d610
Hanbo Cheng
claimed_verified
2024-10-21T07:47:17.268Z
67147c30476f2449e228cad1
/avatars/13976ebe52634c551d41b38da801e35d.svg
Hanbo-Cheng
user
Hanbo-Cheng
67147d177b711cff63c0d611
Limin Lin
admin_assigned
2024-10-21T12:15:07.918Z
6716128fcf2b9210e5cab2db
/avatars/254fa3b1409b844b38e4fd738f638fa6.svg
Limin-Lin
user
Limin-Lin
67147d177b711cff63c0d612
Chenyu Liu
67147d177b711cff63c0d613
Pengcheng Xia
admin_assigned
2024-10-21T12:15:13.738Z
64946adc02a46148e275735c
/avatars/26a20b321f495aa7af1481ba2af392bd.svg
Pengcheng Xia
user
page-xia
67147d177b711cff63c0d614
Pengfei Hu
admin_assigned
2024-10-21T12:15:19.747Z
64291be28136224fee045d36
/avatars/c0d155c053742478e8c6d6bad9e0bd4f.svg
Pengfei Hu
user
Bazhu
67147d177b711cff63c0d615
Jiefeng Ma
67147d177b711cff63c0d616
Jun Du
admin_assigned
2024-10-21T12:15:40.258Z
5e0d54a0fcf41d740b699660
/avatars/67dbb87baf7d75a4ec253cc706f819ce.svg
Tim Du
user
dujun
67147d177b711cff63c0d617
Jia Pan
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework
  for Talking Head Video Generation
Talking head generation intends to produce vivid and realistic talking head
videos from a single portrait and speech audio clip. Although significant
progress has been made in diffusion-based talking head generation, almost all
methods rely on autoregressive strategies, which suffer from limited context
utilization beyond the current generation step, error accumulation, and slower
generation speed. To address these challenges, we present DAWN (Dynamic frame
Avatar With Non-autoregressive diffusion), a framework that enables all-at-once
generation of dynamic-length video sequences. Specifically, it consists of two
main components: (1) audio-driven holistic facial dynamics generation in the
latent motion space, and (2) audio-driven head pose and blink generation.
Extensive experiments demonstrate that our method generates authentic and vivid
videos with precise lip motions, and natural pose/blink movements.
Additionally, with a high generation speed, DAWN possesses strong extrapolation
capabilities, ensuring the stable production of high-quality long videos. These
results highlight the considerable promise and potential impact of DAWN in the
field of talking head video generation. Furthermore, we hope that DAWN sparks
further exploration of non-autoregressive approaches in diffusion models. Our
code will be publicly at https://github.com/Hanbo-Cheng/DAWN-pytorch.
DAWN, a non-autoregressive diffusion framework, generates high-quality, realistic talking head videos with precise lip and natural head movement at high speed.
diffusion-based talking head generation autoregressive strategies latent motion space non-autoregressive diffusion audio-driven holistic facial dynamics audio-driven head pose audio-driven blink generation long videos high-quality videos precise lip motions natural pose/blink movements extrapolation capabilities