Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based
  Spatiotemporal Diffusion for Audio-driven Talking Portrait
Audio-driven single-image talking portrait generation plays a crucial role in
virtual reality, digital human creation, and filmmaking. Existing approaches
are generally categorized into keypoint-based and image-based methods.
Keypoint-based methods effectively preserve character identity but struggle to
capture fine facial details due to the fixed points limitation of the 3D
Morphable Model. Moreover, traditional generative networks face challenges in
establishing causality between audio and keypoints on limited datasets,
resulting in low pose diversity. In contrast, image-based approaches produce
high-quality portraits with diverse details using the diffusion network but
incur identity distortion and expensive computational costs. In this work, we
propose KDTalker, the first framework to combine unsupervised implicit 3D
keypoint with a spatiotemporal diffusion model. Leveraging unsupervised
implicit 3D keypoints, KDTalker adapts facial information densities, allowing
the diffusion process to model diverse head poses and capture fine facial
details flexibly. The custom-designed spatiotemporal attention mechanism
ensures accurate lip synchronization, producing temporally consistent,
high-quality animations while enhancing computational efficiency. Experimental
results demonstrate that KDTalker achieves state-of-the-art performance
regarding lip synchronization accuracy, head pose diversity, and execution
efficiency.Our codes are available at https://github.com/chaolongy/KDTalker.
67da8735cae81b233ffd2927
Chaolong Yang
claimed_verified
2025-03-20T11:00:42.125Z
67da84f499b60bd76c91fd83
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/5iUk_mhBg0lLbUbODEybH.png
Chaolong Yang
user
ChaolongYang
67da8735cae81b233ffd2928
Kai Yao
extracted_pending
2025-03-19T08:58:30.094Z
65d6adbe654f85ff0bb0c246
/avatars/a11b086ae1422b8d64d56d2c5825d3c0.svg
kai yao
user
KaiserYaoJM
67da8735cae81b233ffd2929
Yuyao Yan
admin_assigned
2025-03-20T13:29:13.789Z
673aef01836947961d636a15
/avatars/3d45b5048d7a55a8a31a928081d95d8a.svg
Yuyao Yan
user
Ritayy
67da8735cae81b233ffd292a
Chenru Jiang
admin_assigned
2025-03-20T13:29:20.450Z
6773b0515c88cb6ab34cc8fd
/avatars/5d4c4156463a7d69d88e579bf14215d0.svg
chenru jiang
user
chenrujiang
67da8735cae81b233ffd292b
Weiguang Zhao
claimed_verified
2025-03-20T11:05:14.366Z
672094a12174ca42d583e7bc
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OYcCwVQwR0oDQcVsMr22R.png
Weiguang Zhao
user
weiguangzhao
67da8735cae81b233ffd292c
Jie Sun
67da8735cae81b233ffd292d
Guangliang Cheng
67da8735cae81b233ffd292e
Yifei Zhang
67da8735cae81b233ffd292f
Bin Dong
67da8735cae81b233ffd2930
Kaizhu Huang
Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based
  Spatiotemporal Diffusion for Audio-driven Talking Portrait
Audio-driven single-image talking portrait generation plays a crucial role in
virtual reality, digital human creation, and filmmaking. Existing approaches
are generally categorized into keypoint-based and image-based methods.
Keypoint-based methods effectively preserve character identity but struggle to
capture fine facial details due to the fixed points limitation of the 3D
Morphable Model. Moreover, traditional generative networks face challenges in
establishing causality between audio and keypoints on limited datasets,
resulting in low pose diversity. In contrast, image-based approaches produce
high-quality portraits with diverse details using the diffusion network but
incur identity distortion and expensive computational costs. In this work, we
propose KDTalker, the first framework to combine unsupervised implicit 3D
keypoint with a spatiotemporal diffusion model. Leveraging unsupervised
implicit 3D keypoints, KDTalker adapts facial information densities, allowing
the diffusion process to model diverse head poses and capture fine facial
details flexibly. The custom-designed spatiotemporal attention mechanism
ensures accurate lip synchronization, producing temporally consistent,
high-quality animations while enhancing computational efficiency. Experimental
results demonstrate that KDTalker achieves state-of-the-art performance
regarding lip synchronization accuracy, head pose diversity, and execution
efficiency.Our codes are available at https://github.com/chaolongy/KDTalker.
KDTalker integrates unsupervised implicit 3D keypoints with a spatiotemporal diffusion model to generate high-quality, detail-rich talking portraits with efficient computation and accurate lip synchronization.
3D Morphable Model diffusion network unsupervised implicit 3D keypoints spatiotemporal diffusion model spatiotemporal attention mechanism