title: FantasyTalking: Realistic Talking Portrait Generation via Coherent
    Motion Synthesis
summary: Creating a realistic animatable avatar from a single static portrait remains
  challenging. Existing approaches often struggle to capture subtle facial
  expressions, the associated global body movements, and the dynamic background.
  To address these limitations, we propose a novel framework that leverages a
  pretrained video diffusion transformer model to generate high-fidelity,
  coherent talking portraits with controllable motion dynamics. At the core of
  our work is a dual-stage audio-visual alignment strategy. In the first stage,
  we employ a clip-level training scheme to establish coherent global motion by
  aligning audio-driven dynamics across the entire scene, including the reference
  portrait, contextual objects, and background. In the second stage, we refine
  lip movements at the frame level using a lip-tracing mask, ensuring precise
  synchronization with audio signals. To preserve identity without compromising
  motion flexibility, we replace the commonly used reference network with a
  facial-focused cross-attention module that effectively maintains facial
  consistency throughout the video. Furthermore, we integrate a motion intensity
  modulation module that explicitly controls expression and body motion
  intensity, enabling controllable manipulation of portrait movements beyond mere
  lip motion. Extensive experimental results show that our proposed approach
  achieves higher quality with better realism, coherence, motion intensity, and
  identity preservation. Ours project page:
  https://fantasy-amap.github.io/fantasy-talking/.
paper_authors._id: 67f72ca8353d129fc7bdd504
paper_authors.name: Mengchao Wang
paper_authors._id: 67f72ca8353d129fc7bdd505
paper_authors.name: Qiang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:39:40.647Z
paper_authors._id: 653b195c5f1703225b2fd571
paper_authors.avatarUrl: /avatars/b7f376225cef6c13952c9c5540dd43be.svg
paper_authors.fullname: wangqiang
paper_authors.type: user
paper_authors.user: wangqiang9
paper_authors._id: 67f72ca8353d129fc7bdd506
paper_authors.name: Fan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:50:07.747Z
paper_authors._id: 63048ea19aef62c4013c77aa
paper_authors.avatarUrl: /avatars/b2b2243ccc63cfb5a3289bc2eb1d6293.svg
paper_authors.fullname: fanjiang
paper_authors.type: user
paper_authors.user: fanjiang
paper_authors._id: 67f72ca8353d129fc7bdd507
paper_authors.name: Yaqi Fan
paper_authors._id: 67f72ca8353d129fc7bdd508
paper_authors.name: Yunpeng Zhang
paper_authors._id: 67f72ca8353d129fc7bdd509
paper_authors.name: Yonggang Qi
paper_authors._id: 67f72ca8353d129fc7bdd50a
paper_authors.name: Kun Zhao
paper_authors._id: 67f72ca8353d129fc7bdd50b
paper_authors.name: Mu Xu
paper_title: FantasyTalking: Realistic Talking Portrait Generation via Coherent
    Motion Synthesis
paper_summary: Creating a realistic animatable avatar from a single static portrait remains
  challenging. Existing approaches often struggle to capture subtle facial
  expressions, the associated global body movements, and the dynamic background.
  To address these limitations, we propose a novel framework that leverages a
  pretrained video diffusion transformer model to generate high-fidelity,
  coherent talking portraits with controllable motion dynamics. At the core of
  our work is a dual-stage audio-visual alignment strategy. In the first stage,
  we employ a clip-level training scheme to establish coherent global motion by
  aligning audio-driven dynamics across the entire scene, including the reference
  portrait, contextual objects, and background. In the second stage, we refine
  lip movements at the frame level using a lip-tracing mask, ensuring precise
  synchronization with audio signals. To preserve identity without compromising
  motion flexibility, we replace the commonly used reference network with a
  facial-focused cross-attention module that effectively maintains facial
  consistency throughout the video. Furthermore, we integrate a motion intensity
  modulation module that explicitly controls expression and body motion
  intensity, enabling controllable manipulation of portrait movements beyond mere
  lip motion. Extensive experimental results show that our proposed approach
  achieves higher quality with better realism, coherence, motion intensity, and
  identity preservation. Ours project page:
  https://fantasy-amap.github.io/fantasy-talking/.
paper_ai_summary: A novel framework uses pretrained video diffusion transformer models to generate realistic, high-fidelity talking portraits with controlled motion dynamics, achieving better realism and identity preservation.
paper_ai_keywords: video diffusion transformer dual-stage audio-visual alignment clip-level training frame-level refinement lip-tracing mask facial-focused cross-attention motion intensity modulation facial consistency