MoCha: Towards Movie-Grade Talking Character Synthesis
Recent advancements in video generation have achieved impressive motion
realism, yet they often overlook character-driven storytelling, a crucial task
for automated film, animation generation. We introduce Talking Characters, a
more realistic task to generate talking character animations directly from
speech and text. Unlike talking head, Talking Characters aims at generating the
full portrait of one or more characters beyond the facial region. In this
paper, we propose MoCha, the first of its kind to generate talking characters.
To ensure precise synchronization between video and speech, we propose a
speech-video window attention mechanism that effectively aligns speech and
video tokens. To address the scarcity of large-scale speech-labeled video
datasets, we introduce a joint training strategy that leverages both
speech-labeled and text-labeled video data, significantly improving
generalization across diverse character actions. We also design structured
prompt templates with character tags, enabling, for the first time,
multi-character conversation with turn-based dialogue-allowing AI-generated
characters to engage in context-aware conversations with cinematic coherence.
Extensive qualitative and quantitative evaluations, including human preference
studies and benchmark comparisons, demonstrate that MoCha sets a new standard
for AI-generated cinematic storytelling, achieving superior realism,
expressiveness, controllability and generalization.
67eb4bd0eca57c4eebbb343a
Cong Wei
claimed_verified
2025-04-01T07:47:21.554Z
64f8e358766ff9f3d2b0de84
https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
Cong Wei
user
CongWei1230
67eb4bd0eca57c4eebbb343b
Bo Sun
67eb4bd0eca57c4eebbb343c
Haoyu Ma
admin_assigned
2025-04-01T07:54:47.847Z
650a8979c19e5b4c8a6ff062
https://cdn-avatars.huggingface.co/v1/production/uploads/650a8979c19e5b4c8a6ff062/64_JuECX_k_-uK7m7nlua.jpeg
Haoyu Ma
user
haoyum1997
67eb4bd0eca57c4eebbb343d
Ji Hou
claimed_verified
2025-04-02T08:51:24.595Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
67eb4bd0eca57c4eebbb343e
Felix Juefei-Xu
admin_assigned
2025-04-01T07:55:03.326Z
6444e8911cfc9ae6bb3ad216
/avatars/8c06e064cf24789e4131f7af06dac86b.svg
Xu
user
FelixXu
67eb4bd0eca57c4eebbb343f
Zecheng He
67eb4bd0eca57c4eebbb3440
Xiaoliang Dai
admin_assigned
2025-04-01T07:54:01.490Z
6549417b3ce45eb764faf993
/avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
Xiaoliang Dai
user
daixl1992
67eb4bd0eca57c4eebbb3441
Luxin Zhang
claimed_verified
2025-04-02T08:51:22.257Z
67ecad336560da48c5c2bf08
/avatars/5ef8381504bfc226b0d82c5a36f04ca3.svg
Luxin Zhang
user
luxinzhang
67eb4bd0eca57c4eebbb3442
Kunpeng Li
67eb4bd0eca57c4eebbb3443
Tingbo Hou
admin_assigned
2025-04-01T07:53:10.214Z
655846d7ed8df83128f5826a
/avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
Hou
user
Tingbo
67eb4bd0eca57c4eebbb3444
Animesh Sinha
claimed_verified
2025-04-02T08:23:47.875Z
63772d28e069bdcdbb8e4a29
/avatars/be71b00e009f2093a3ff79f07815e68e.svg
Animesh Sinha
user
animeshsinha
67eb4bd0eca57c4eebbb3445
Peter Vajda
67eb4bd0eca57c4eebbb3446
Wenhu Chen
admin_assigned
2025-04-01T07:52:50.248Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
MoCha: Towards Movie-Grade Talking Character Synthesis
Recent advancements in video generation have achieved impressive motion
realism, yet they often overlook character-driven storytelling, a crucial task
for automated film, animation generation. We introduce Talking Characters, a
more realistic task to generate talking character animations directly from
speech and text. Unlike talking head, Talking Characters aims at generating the
full portrait of one or more characters beyond the facial region. In this
paper, we propose MoCha, the first of its kind to generate talking characters.
To ensure precise synchronization between video and speech, we propose a
speech-video window attention mechanism that effectively aligns speech and
video tokens. To address the scarcity of large-scale speech-labeled video
datasets, we introduce a joint training strategy that leverages both
speech-labeled and text-labeled video data, significantly improving
generalization across diverse character actions. We also design structured
prompt templates with character tags, enabling, for the first time,
multi-character conversation with turn-based dialogue-allowing AI-generated
characters to engage in context-aware conversations with cinematic coherence.
Extensive qualitative and quantitative evaluations, including human preference
studies and benchmark comparisons, demonstrate that MoCha sets a new standard
for AI-generated cinematic storytelling, achieving superior realism,
expressiveness, controllability and generalization.
MoCha generates realistic talking character animations from speech and text using a speech-video attention mechanism and joint training on speech-labeled and text-labeled data, enabling multi-character conversations and superior realism.
MoCha speech-video window attention mechanism joint training strategy structured prompt templates multi-character conversation cinematic coherence