title: MoCha: Towards Movie-Grade Talking Character Synthesis
summary: Recent advancements in video generation have achieved impressive motion
  realism, yet they often overlook character-driven storytelling, a crucial task
  for automated film, animation generation. We introduce Talking Characters, a
  more realistic task to generate talking character animations directly from
  speech and text. Unlike talking head, Talking Characters aims at generating the
  full portrait of one or more characters beyond the facial region. In this
  paper, we propose MoCha, the first of its kind to generate talking characters.
  To ensure precise synchronization between video and speech, we propose a
  speech-video window attention mechanism that effectively aligns speech and
  video tokens. To address the scarcity of large-scale speech-labeled video
  datasets, we introduce a joint training strategy that leverages both
  speech-labeled and text-labeled video data, significantly improving
  generalization across diverse character actions. We also design structured
  prompt templates with character tags, enabling, for the first time,
  multi-character conversation with turn-based dialogue-allowing AI-generated
  characters to engage in context-aware conversations with cinematic coherence.
  Extensive qualitative and quantitative evaluations, including human preference
  studies and benchmark comparisons, demonstrate that MoCha sets a new standard
  for AI-generated cinematic storytelling, achieving superior realism,
  expressiveness, controllability and generalization.
paper_authors._id: 67eb4bd0eca57c4eebbb343a
paper_authors.name: Cong Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:47:21.554Z
paper_authors._id: 64f8e358766ff9f3d2b0de84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
paper_authors.fullname: Cong Wei
paper_authors.type: user
paper_authors.user: CongWei1230
paper_authors._id: 67eb4bd0eca57c4eebbb343b
paper_authors.name: Bo Sun
paper_authors._id: 67eb4bd0eca57c4eebbb343c
paper_authors.name: Haoyu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T07:54:47.847Z
paper_authors._id: 650a8979c19e5b4c8a6ff062
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650a8979c19e5b4c8a6ff062/64_JuECX_k_-uK7m7nlua.jpeg
paper_authors.fullname: Haoyu Ma
paper_authors.type: user
paper_authors.user: haoyum1997
paper_authors._id: 67eb4bd0eca57c4eebbb343d
paper_authors.name: Ji Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:51:24.595Z
paper_authors._id: 63084636f0dc38fb47c2dc6b
paper_authors.avatarUrl: /avatars/04edff10a048b5d93981e6f5af3ecf40.svg
paper_authors.fullname: Ji Hou
paper_authors.type: user
paper_authors.user: jhou90
paper_authors._id: 67eb4bd0eca57c4eebbb343e
paper_authors.name: Felix Juefei-Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T07:55:03.326Z
paper_authors._id: 6444e8911cfc9ae6bb3ad216
paper_authors.avatarUrl: /avatars/8c06e064cf24789e4131f7af06dac86b.svg
paper_authors.fullname: Xu
paper_authors.type: user
paper_authors.user: FelixXu
paper_authors._id: 67eb4bd0eca57c4eebbb343f
paper_authors.name: Zecheng He
paper_authors._id: 67eb4bd0eca57c4eebbb3440
paper_authors.name: Xiaoliang Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T07:54:01.490Z
paper_authors._id: 6549417b3ce45eb764faf993
paper_authors.avatarUrl: /avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
paper_authors.fullname: Xiaoliang Dai
paper_authors.type: user
paper_authors.user: daixl1992
paper_authors._id: 67eb4bd0eca57c4eebbb3441
paper_authors.name: Luxin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:51:22.257Z
paper_authors._id: 67ecad336560da48c5c2bf08
paper_authors.avatarUrl: /avatars/5ef8381504bfc226b0d82c5a36f04ca3.svg
paper_authors.fullname: Luxin Zhang
paper_authors.type: user
paper_authors.user: luxinzhang
paper_authors._id: 67eb4bd0eca57c4eebbb3442
paper_authors.name: Kunpeng Li
paper_authors._id: 67eb4bd0eca57c4eebbb3443
paper_authors.name: Tingbo Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T07:53:10.214Z
paper_authors._id: 655846d7ed8df83128f5826a
paper_authors.avatarUrl: /avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
paper_authors.fullname: Hou
paper_authors.type: user
paper_authors.user: Tingbo
paper_authors._id: 67eb4bd0eca57c4eebbb3444
paper_authors.name: Animesh Sinha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:23:47.875Z
paper_authors._id: 63772d28e069bdcdbb8e4a29
paper_authors.avatarUrl: /avatars/be71b00e009f2093a3ff79f07815e68e.svg
paper_authors.fullname: Animesh Sinha
paper_authors.type: user
paper_authors.user: animeshsinha
paper_authors._id: 67eb4bd0eca57c4eebbb3445
paper_authors.name: Peter Vajda
paper_authors._id: 67eb4bd0eca57c4eebbb3446
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T07:52:50.248Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: MoCha: Towards Movie-Grade Talking Character Synthesis
paper_summary: Recent advancements in video generation have achieved impressive motion
  realism, yet they often overlook character-driven storytelling, a crucial task
  for automated film, animation generation. We introduce Talking Characters, a
  more realistic task to generate talking character animations directly from
  speech and text. Unlike talking head, Talking Characters aims at generating the
  full portrait of one or more characters beyond the facial region. In this
  paper, we propose MoCha, the first of its kind to generate talking characters.
  To ensure precise synchronization between video and speech, we propose a
  speech-video window attention mechanism that effectively aligns speech and
  video tokens. To address the scarcity of large-scale speech-labeled video
  datasets, we introduce a joint training strategy that leverages both
  speech-labeled and text-labeled video data, significantly improving
  generalization across diverse character actions. We also design structured
  prompt templates with character tags, enabling, for the first time,
  multi-character conversation with turn-based dialogue-allowing AI-generated
  characters to engage in context-aware conversations with cinematic coherence.
  Extensive qualitative and quantitative evaluations, including human preference
  studies and benchmark comparisons, demonstrate that MoCha sets a new standard
  for AI-generated cinematic storytelling, achieving superior realism,
  expressiveness, controllability and generalization.
paper_ai_summary: MoCha generates realistic talking character animations from speech and text using a speech-video attention mechanism and joint training on speech-labeled and text-labeled data, enabling multi-character conversations and superior realism.
paper_ai_keywords: MoCha speech-video window attention mechanism joint training strategy structured prompt templates multi-character conversation cinematic coherence