FantasyTalking: Realistic Talking Portrait Generation via Coherent
  Motion Synthesis
Creating a realistic animatable avatar from a single static portrait remains
challenging. Existing approaches often struggle to capture subtle facial
expressions, the associated global body movements, and the dynamic background.
To address these limitations, we propose a novel framework that leverages a
pretrained video diffusion transformer model to generate high-fidelity,
coherent talking portraits with controllable motion dynamics. At the core of
our work is a dual-stage audio-visual alignment strategy. In the first stage,
we employ a clip-level training scheme to establish coherent global motion by
aligning audio-driven dynamics across the entire scene, including the reference
portrait, contextual objects, and background. In the second stage, we refine
lip movements at the frame level using a lip-tracing mask, ensuring precise
synchronization with audio signals. To preserve identity without compromising
motion flexibility, we replace the commonly used reference network with a
facial-focused cross-attention module that effectively maintains facial
consistency throughout the video. Furthermore, we integrate a motion intensity
modulation module that explicitly controls expression and body motion
intensity, enabling controllable manipulation of portrait movements beyond mere
lip motion. Extensive experimental results show that our proposed approach
achieves higher quality with better realism, coherence, motion intensity, and
identity preservation. Ours project page:
https://fantasy-amap.github.io/fantasy-talking/.
67f72ca8353d129fc7bdd504
Mengchao Wang
67f72ca8353d129fc7bdd505
Qiang Wang
claimed_verified
2025-04-10T06:39:40.647Z
653b195c5f1703225b2fd571
/avatars/b7f376225cef6c13952c9c5540dd43be.svg
wangqiang
user
wangqiang9
67f72ca8353d129fc7bdd506
Fan Jiang
admin_assigned
2025-04-10T08:50:07.747Z
63048ea19aef62c4013c77aa
/avatars/b2b2243ccc63cfb5a3289bc2eb1d6293.svg
fanjiang
user
fanjiang
67f72ca8353d129fc7bdd507
Yaqi Fan
67f72ca8353d129fc7bdd508
Yunpeng Zhang
67f72ca8353d129fc7bdd509
Yonggang Qi
67f72ca8353d129fc7bdd50a
Kun Zhao
67f72ca8353d129fc7bdd50b
Mu Xu
FantasyTalking: Realistic Talking Portrait Generation via Coherent
  Motion Synthesis
Creating a realistic animatable avatar from a single static portrait remains
challenging. Existing approaches often struggle to capture subtle facial
expressions, the associated global body movements, and the dynamic background.
To address these limitations, we propose a novel framework that leverages a
pretrained video diffusion transformer model to generate high-fidelity,
coherent talking portraits with controllable motion dynamics. At the core of
our work is a dual-stage audio-visual alignment strategy. In the first stage,
we employ a clip-level training scheme to establish coherent global motion by
aligning audio-driven dynamics across the entire scene, including the reference
portrait, contextual objects, and background. In the second stage, we refine
lip movements at the frame level using a lip-tracing mask, ensuring precise
synchronization with audio signals. To preserve identity without compromising
motion flexibility, we replace the commonly used reference network with a
facial-focused cross-attention module that effectively maintains facial
consistency throughout the video. Furthermore, we integrate a motion intensity
modulation module that explicitly controls expression and body motion
intensity, enabling controllable manipulation of portrait movements beyond mere
lip motion. Extensive experimental results show that our proposed approach
achieves higher quality with better realism, coherence, motion intensity, and
identity preservation. Ours project page:
https://fantasy-amap.github.io/fantasy-talking/.
A novel framework uses pretrained video diffusion transformer models to generate realistic, high-fidelity talking portraits with controlled motion dynamics, achieving better realism and identity preservation.
video diffusion transformer dual-stage audio-visual alignment clip-level training frame-level refinement lip-tracing mask facial-focused cross-attention motion intensity modulation facial consistency