title: Compositional 3D-aware Video Generation with LLM Director
summary: Significant progress has been made in text-to-video generation through the
  use of powerful generative models and large-scale internet data. However,
  substantial challenges remain in precisely controlling individual concepts
  within the generated video, such as the motion and appearance of specific
  characters and the movement of viewpoints. In this work, we propose a novel
  paradigm that generates each concept in 3D representation separately and then
  composes them with priors from Large Language Models (LLM) and 2D diffusion
  models. Specifically, given an input textual prompt, our scheme consists of
  three stages: 1) We leverage LLM as the director to first decompose the complex
  query into several sub-prompts that indicate individual concepts within the
  video~(e.g., scene, objects, motions), then we let LLM to invoke
  pre-trained expert models to obtain corresponding 3D representations of
  concepts. 2) To compose these representations, we prompt multi-modal LLM to
  produce coarse guidance on the scales and coordinates of trajectories for the
  objects. 3) To make the generated frames adhere to natural image distribution,
  we further leverage 2D diffusion priors and use Score Distillation Sampling to
  refine the composition. Extensive experiments demonstrate that our method can
  generate high-fidelity videos from text with diverse motion and flexible
  control over each concept. Project page: https://aka.ms/c3v.
paper_authors._id: 66d7ef327e6c9509bb09f617
paper_authors.name: Hanxin Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:15:00.377Z
paper_authors._id: 650e5beff874d950df2203f0
paper_authors.avatarUrl: /avatars/384b35ea59b55f8d454be9efb7cce47c.svg
paper_authors.fullname: Hanxin ZHU
paper_authors.type: user
paper_authors.user: ingzzzz
paper_authors._id: 66d7ef327e6c9509bb09f618
paper_authors.name: Tianyu He
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-04T05:28:42.906Z
paper_authors._id: 619b7b1cab4c7b7f16a7d59e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
paper_authors.fullname: Tianyu He
paper_authors.type: user
paper_authors.user: deeptimhe
paper_authors._id: 66d7ef327e6c9509bb09f619
paper_authors.name: Anni Tang
paper_authors._id: 66d7ef327e6c9509bb09f61a
paper_authors.name: Junliang Guo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-04T05:25:07.682Z
paper_authors._id: 66868659ccb9539da85c4e14
paper_authors.avatarUrl: /avatars/515a49363872c23d57a6f75063606348.svg
paper_authors.fullname: Junliang Guo
paper_authors.type: user
paper_authors.user: leo-guo
paper_authors._id: 66d7ef327e6c9509bb09f61b
paper_authors.name: Zhibo Chen
paper_authors._id: 66d7ef327e6c9509bb09f61c
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:38:09.562Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_title: Compositional 3D-aware Video Generation with LLM Director
paper_summary: Significant progress has been made in text-to-video generation through the
  use of powerful generative models and large-scale internet data. However,
  substantial challenges remain in precisely controlling individual concepts
  within the generated video, such as the motion and appearance of specific
  characters and the movement of viewpoints. In this work, we propose a novel
  paradigm that generates each concept in 3D representation separately and then
  composes them with priors from Large Language Models (LLM) and 2D diffusion
  models. Specifically, given an input textual prompt, our scheme consists of
  three stages: 1) We leverage LLM as the director to first decompose the complex
  query into several sub-prompts that indicate individual concepts within the
  video~(e.g., scene, objects, motions), then we let LLM to invoke
  pre-trained expert models to obtain corresponding 3D representations of
  concepts. 2) To compose these representations, we prompt multi-modal LLM to
  produce coarse guidance on the scales and coordinates of trajectories for the
  objects. 3) To make the generated frames adhere to natural image distribution,
  we further leverage 2D diffusion priors and use Score Distillation Sampling to
  refine the composition. Extensive experiments demonstrate that our method can
  generate high-fidelity videos from text with diverse motion and flexible
  control over each concept. Project page: https://aka.ms/c3v.
paper_ai_summary: A novel text-to-video generation method decomposes concepts into 3D representations using LLMs, composes them with multi-modal LLM guidance, and refines with 2D diffusion models to produce high-fidelity videos with controlled motion and flexibility.
paper_ai_keywords: generative models large-scale internet data text-to-video generation Large Language Models LLM 3D representation 2D diffusion models Score Distillation Sampling