title: GenMAC: Compositional Text-to-Video Generation with Multi-Agent
    Collaboration
summary: Text-to-video generation models have shown significant progress in the recent
  years. However, they still struggle with generating complex dynamic scenes
  based on compositional text prompts, such as attribute binding for multiple
  objects, temporal dynamics associated with different objects, and interactions
  between objects. Our key motivation is that complex tasks can be decomposed
  into simpler ones, each handled by a role-specialized MLLM agent. Multiple
  agents can collaborate together to achieve collective intelligence for complex
  goals. We propose GenMAC, an iterative, multi-agent framework that enables
  compositional text-to-video generation. The collaborative workflow includes
  three stages: Design, Generation, and Redesign, with an iterative loop between
  the Generation and Redesign stages to progressively verify and refine the
  generated videos. The Redesign stage is the most challenging stage that aims to
  verify the generated videos, suggest corrections, and redesign the text
  prompts, frame-wise layouts, and guidance scales for the next iteration of
  generation. To avoid hallucination of a single MLLM agent, we decompose this
  stage to four sequentially-executed MLLM-based agents: verification agent,
  suggestion agent, correction agent, and output structuring agent. Furthermore,
  to tackle diverse scenarios of compositional text-to-video generation, we
  design a self-routing mechanism to adaptively select the proper correction
  agent from a collection of correction agents each specialized for one scenario.
  Extensive experiments demonstrate the effectiveness of GenMAC, achieving
  state-of-the art performance in compositional text-to-video generation.
paper_authors._id: 6752a1062f16c25ac96879dc
paper_authors.name: Kaiyi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:48:59.205Z
paper_authors._id: 637cba13b8e573d75be96ea6
paper_authors.avatarUrl: /avatars/5eca230e63d66947b2a05c1ff964a96c.svg
paper_authors.fullname: Nina
paper_authors.type: user
paper_authors.user: NinaKarine
paper_authors._id: 6752a1062f16c25ac96879dd
paper_authors.name: Yukun Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T14:34:31.663Z
paper_authors._id: 638ee900ee7e45e0474a5712
paper_authors.avatarUrl: /avatars/eadb5ae2fc92bd9af5516acbc8f1bdf0.svg
paper_authors.fullname: Yukun Huang
paper_authors.type: user
paper_authors.user: KevinHuang
paper_authors._id: 6752a1062f16c25ac96879de
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:37:18.692Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 6752a1062f16c25ac96879df
paper_authors.name: Zinan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:07:13.931Z
paper_authors._id: 64c832a8c547ed5243d29630
paper_authors.avatarUrl: /avatars/59d1975634e84095b69423c02441d453.svg
paper_authors.fullname: Zinan Lin
paper_authors.type: user
paper_authors.user: fjxmlzn
paper_authors._id: 6752a1062f16c25ac96879e0
paper_authors.name: Yu Wang
paper_authors._id: 6752a1062f16c25ac96879e1
paper_authors.name: Xihui Liu
paper_title: GenMAC: Compositional Text-to-Video Generation with Multi-Agent
    Collaboration
paper_summary: Text-to-video generation models have shown significant progress in the recent
  years. However, they still struggle with generating complex dynamic scenes
  based on compositional text prompts, such as attribute binding for multiple
  objects, temporal dynamics associated with different objects, and interactions
  between objects. Our key motivation is that complex tasks can be decomposed
  into simpler ones, each handled by a role-specialized MLLM agent. Multiple
  agents can collaborate together to achieve collective intelligence for complex
  goals. We propose GenMAC, an iterative, multi-agent framework that enables
  compositional text-to-video generation. The collaborative workflow includes
  three stages: Design, Generation, and Redesign, with an iterative loop between
  the Generation and Redesign stages to progressively verify and refine the
  generated videos. The Redesign stage is the most challenging stage that aims to
  verify the generated videos, suggest corrections, and redesign the text
  prompts, frame-wise layouts, and guidance scales for the next iteration of
  generation. To avoid hallucination of a single MLLM agent, we decompose this
  stage to four sequentially-executed MLLM-based agents: verification agent,
  suggestion agent, correction agent, and output structuring agent. Furthermore,
  to tackle diverse scenarios of compositional text-to-video generation, we
  design a self-routing mechanism to adaptively select the proper correction
  agent from a collection of correction agents each specialized for one scenario.
  Extensive experiments demonstrate the effectiveness of GenMAC, achieving
  state-of-the art performance in compositional text-to-video generation.
paper_ai_keywords: text-to-video generation compositional text prompts attribute binding temporal dynamics interactions between objects role-specialized MLLM agent multi-agent framework GenMAC iterative workflow Design Generation Redesign collaborative workflow verification agent suggestion agent correction agent output structuring agent self-routing mechanism state-of-the-art performance