title: MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
summary: Recent advances in video diffusion models have unlocked new potential for
  realistic audio-driven talking video generation. However, achieving seamless
  audio-lip synchronization, maintaining long-term identity consistency, and
  producing natural, audio-aligned expressions in generated talking videos remain
  significant challenges. To address these challenges, we propose Memory-guided
  EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation
  approach to generate identity-consistent and expressive talking videos. Our
  approach is built around two key modules: (1) a memory-guided temporal module,
  which enhances long-term identity consistency and motion smoothness by
  developing memory states to store information from a longer past context to
  guide temporal modeling via linear attention; and (2) an emotion-aware audio
  module, which replaces traditional cross attention with multi-modal attention
  to enhance audio-video interaction, while detecting emotions from audio to
  refine facial expressions via emotion adaptive layer norm. Extensive
  quantitative and qualitative results demonstrate that MEMO generates more
  realistic talking videos across diverse image and audio types, outperforming
  state-of-the-art methods in overall quality, audio-lip synchronization,
  identity consistency, and expression-emotion alignment.
paper_authors._id: 675265f58be943d9cb551721
paper_authors.name: Longtao Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T10:01:44.427Z
paper_authors._id: 63db5dc49f2687298a1547bf
paper_authors.avatarUrl: /avatars/a3f87dbf41b836f043f01b61ffd820ea.svg
paper_authors.fullname: Longtao Zheng
paper_authors.type: user
paper_authors.user: ltzheng
paper_authors._id: 675265f58be943d9cb551722
paper_authors.name: Yifan Zhang
paper_authors._id: 675265f58be943d9cb551723
paper_authors.name: Hanzhong Guo
paper_authors._id: 675265f58be943d9cb551724
paper_authors.name: Jiachun Pan
paper_authors._id: 675265f58be943d9cb551725
paper_authors.name: Zhenxiong Tan
paper_authors._id: 675265f58be943d9cb551726
paper_authors.name: Jiahao Lu
paper_authors._id: 675265f58be943d9cb551727
paper_authors.name: Chuanxin Tang
paper_authors._id: 675265f58be943d9cb551728
paper_authors.name: Bo An
paper_authors._id: 675265f58be943d9cb551729
paper_authors.name: Shuicheng Yan
paper_title: MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
paper_summary: Recent advances in video diffusion models have unlocked new potential for
  realistic audio-driven talking video generation. However, achieving seamless
  audio-lip synchronization, maintaining long-term identity consistency, and
  producing natural, audio-aligned expressions in generated talking videos remain
  significant challenges. To address these challenges, we propose Memory-guided
  EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation
  approach to generate identity-consistent and expressive talking videos. Our
  approach is built around two key modules: (1) a memory-guided temporal module,
  which enhances long-term identity consistency and motion smoothness by
  developing memory states to store information from a longer past context to
  guide temporal modeling via linear attention; and (2) an emotion-aware audio
  module, which replaces traditional cross attention with multi-modal attention
  to enhance audio-video interaction, while detecting emotions from audio to
  refine facial expressions via emotion adaptive layer norm. Extensive
  quantitative and qualitative results demonstrate that MEMO generates more
  realistic talking videos across diverse image and audio types, outperforming
  state-of-the-art methods in overall quality, audio-lip synchronization,
  identity consistency, and expression-emotion alignment.
paper_ai_keywords: video diffusion models audio-driven talking video generation audio-lip synchronization identity consistency memory-guided EMotion-aware diffusion (MEMO) memory-guided temporal module long-term identity consistency motion smoothness memory states linear attention emotion-aware audio module multi-modal attention audio-video interaction emotion adaptive layer realistic talking videos expression-emotion alignment