title: CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance
summary: Video generation has witnessed remarkable progress with the advent of deep
  generative models, particularly diffusion models. While existing methods excel
  in generating high-quality videos from text prompts or single images,
  personalized multi-subject video generation remains a largely unexplored
  challenge. This task involves synthesizing videos that incorporate multiple
  distinct subjects, each defined by separate reference images, while ensuring
  temporal and spatial consistency. Current approaches primarily rely on mapping
  subject images to keywords in text prompts, which introduces ambiguity and
  limits their ability to model subject relationships effectively. In this paper,
  we propose CINEMA, a novel framework for coherent multi-subject video
  generation by leveraging Multimodal Large Language Model (MLLM). Our approach
  eliminates the need for explicit correspondences between subject images and
  text entities, mitigating ambiguity and reducing annotation effort. By
  leveraging MLLM to interpret subject relationships, our method facilitates
  scalability, enabling the use of large and diverse datasets for training.
  Furthermore, our framework can be conditioned on varying numbers of subjects,
  offering greater flexibility in personalized content creation. Through
  extensive evaluations, we demonstrate that our approach significantly improves
  subject consistency, and overall video coherence, paving the way for advanced
  applications in storytelling, interactive media, and personalized video
  generation.
paper_authors._id: 67d39679ea264394acf948ad
paper_authors.name: Yufan Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:08:35.798Z
paper_authors._id: 64210d1fd039a891a914986d
paper_authors.avatarUrl: /avatars/b178a768657eb223bdbfbd9e0a2000ff.svg
paper_authors.fullname: Yufan Deng
paper_authors.type: user
paper_authors.user: dyf
paper_authors._id: 67d39679ea264394acf948ae
paper_authors.name: Xun Guo
paper_authors._id: 67d39679ea264394acf948af
paper_authors.name: Yizhi Wang
paper_authors._id: 67d39679ea264394acf948b0
paper_authors.name: Jacob Zhiyuan Fang
paper_authors._id: 67d39679ea264394acf948b1
paper_authors.name: Angtian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:09:20.695Z
paper_authors._id: 66a800a2c1454e2221e77473
paper_authors.avatarUrl: /avatars/03cd7a4294f82168b4ca0541aadcb67b.svg
paper_authors.fullname: Angtian Wang
paper_authors.type: user
paper_authors.user: angtian
paper_authors._id: 67d39679ea264394acf948b2
paper_authors.name: Shenghai Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T08:56:29.782Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 67d39679ea264394acf948b3
paper_authors.name: Yiding Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:09:14.383Z
paper_authors._id: 672ce28bb4215fd38821ed9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ruFQmhLESJ4j9cxh6sW_J.png
paper_authors.fullname: Yiding Yang
paper_authors.type: user
paper_authors.user: u302117
paper_authors._id: 67d39679ea264394acf948b4
paper_authors.name: Bo Liu
paper_authors._id: 67d39679ea264394acf948b5
paper_authors.name: Haibin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:09:43.105Z
paper_authors._id: 66624861b01c3b590a8c289b
paper_authors.avatarUrl: /avatars/afd02857eb1632680d0d53da4488c8bc.svg
paper_authors.fullname: Haibin Huang
paper_authors.type: user
paper_authors.user: brotherhuang
paper_authors._id: 67d39679ea264394acf948b6
paper_authors.name: Chongyang Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:09:50.116Z
paper_authors._id: 66fedabfa77ec468ff320767
paper_authors.avatarUrl: /avatars/5bd869c30df0f71733006081343b1bd0.svg
paper_authors.fullname: Chongyang Ma
paper_authors.type: user
paper_authors.user: chongyangma
paper_title: CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance
paper_summary: Video generation has witnessed remarkable progress with the advent of deep
  generative models, particularly diffusion models. While existing methods excel
  in generating high-quality videos from text prompts or single images,
  personalized multi-subject video generation remains a largely unexplored
  challenge. This task involves synthesizing videos that incorporate multiple
  distinct subjects, each defined by separate reference images, while ensuring
  temporal and spatial consistency. Current approaches primarily rely on mapping
  subject images to keywords in text prompts, which introduces ambiguity and
  limits their ability to model subject relationships effectively. In this paper,
  we propose CINEMA, a novel framework for coherent multi-subject video
  generation by leveraging Multimodal Large Language Model (MLLM). Our approach
  eliminates the need for explicit correspondences between subject images and
  text entities, mitigating ambiguity and reducing annotation effort. By
  leveraging MLLM to interpret subject relationships, our method facilitates
  scalability, enabling the use of large and diverse datasets for training.
  Furthermore, our framework can be conditioned on varying numbers of subjects,
  offering greater flexibility in personalized content creation. Through
  extensive evaluations, we demonstrate that our approach significantly improves
  subject consistency, and overall video coherence, paving the way for advanced
  applications in storytelling, interactive media, and personalized video
  generation.
paper_ai_keywords: diffusion models multimodal large language model (MLLM)