CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance
Video generation has witnessed remarkable progress with the advent of deep
generative models, particularly diffusion models. While existing methods excel
in generating high-quality videos from text prompts or single images,
personalized multi-subject video generation remains a largely unexplored
challenge. This task involves synthesizing videos that incorporate multiple
distinct subjects, each defined by separate reference images, while ensuring
temporal and spatial consistency. Current approaches primarily rely on mapping
subject images to keywords in text prompts, which introduces ambiguity and
limits their ability to model subject relationships effectively. In this paper,
we propose CINEMA, a novel framework for coherent multi-subject video
generation by leveraging Multimodal Large Language Model (MLLM). Our approach
eliminates the need for explicit correspondences between subject images and
text entities, mitigating ambiguity and reducing annotation effort. By
leveraging MLLM to interpret subject relationships, our method facilitates
scalability, enabling the use of large and diverse datasets for training.
Furthermore, our framework can be conditioned on varying numbers of subjects,
offering greater flexibility in personalized content creation. Through
extensive evaluations, we demonstrate that our approach significantly improves
subject consistency, and overall video coherence, paving the way for advanced
applications in storytelling, interactive media, and personalized video
generation.
67d39679ea264394acf948ad
Yufan Deng
admin_assigned
2025-03-14T11:08:35.798Z
64210d1fd039a891a914986d
/avatars/b178a768657eb223bdbfbd9e0a2000ff.svg
Yufan Deng
user
dyf
67d39679ea264394acf948ae
Xun Guo
67d39679ea264394acf948af
Yizhi Wang
67d39679ea264394acf948b0
Jacob Zhiyuan Fang
67d39679ea264394acf948b1
Angtian Wang
admin_assigned
2025-03-14T11:09:20.695Z
66a800a2c1454e2221e77473
/avatars/03cd7a4294f82168b4ca0541aadcb67b.svg
Angtian Wang
user
angtian
67d39679ea264394acf948b2
Shenghai Yuan
claimed_verified
2025-03-14T08:56:29.782Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
user
BestWishYsh
67d39679ea264394acf948b3
Yiding Yang
admin_assigned
2025-03-14T11:09:14.383Z
672ce28bb4215fd38821ed9a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ruFQmhLESJ4j9cxh6sW_J.png
Yiding Yang
user
u302117
67d39679ea264394acf948b4
Bo Liu
67d39679ea264394acf948b5
Haibin Huang
admin_assigned
2025-03-14T11:09:43.105Z
66624861b01c3b590a8c289b
/avatars/afd02857eb1632680d0d53da4488c8bc.svg
Haibin Huang
user
brotherhuang
67d39679ea264394acf948b6
Chongyang Ma
admin_assigned
2025-03-14T11:09:50.116Z
66fedabfa77ec468ff320767
/avatars/5bd869c30df0f71733006081343b1bd0.svg
Chongyang Ma
user
chongyangma
CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance
Video generation has witnessed remarkable progress with the advent of deep
generative models, particularly diffusion models. While existing methods excel
in generating high-quality videos from text prompts or single images,
personalized multi-subject video generation remains a largely unexplored
challenge. This task involves synthesizing videos that incorporate multiple
distinct subjects, each defined by separate reference images, while ensuring
temporal and spatial consistency. Current approaches primarily rely on mapping
subject images to keywords in text prompts, which introduces ambiguity and
limits their ability to model subject relationships effectively. In this paper,
we propose CINEMA, a novel framework for coherent multi-subject video
generation by leveraging Multimodal Large Language Model (MLLM). Our approach
eliminates the need for explicit correspondences between subject images and
text entities, mitigating ambiguity and reducing annotation effort. By
leveraging MLLM to interpret subject relationships, our method facilitates
scalability, enabling the use of large and diverse datasets for training.
Furthermore, our framework can be conditioned on varying numbers of subjects,
offering greater flexibility in personalized content creation. Through
extensive evaluations, we demonstrate that our approach significantly improves
subject consistency, and overall video coherence, paving the way for advanced
applications in storytelling, interactive media, and personalized video
generation.
CINEMA uses Multimodal Large Language Model (MLLM) to generate coherent videos with multiple subjects, improving consistency and flexibility.
diffusion models Multimodal Large Language Model MLLM coherent video generation subject consistency video coherence