title: VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided
    Planning
summary: Although recent text-to-video (T2V) generation methods have seen significant
  advancements, most of these works focus on producing short video clips of a
  single event with a single background (i.e., single-scene videos). Meanwhile,
  recent large language models (LLMs) have demonstrated their capability in
  generating layouts and programs to control downstream visual modules such as
  image generation models. This raises an important question: can we leverage the
  knowledge embedded in these LLMs for temporally consistent long video
  generation? In this paper, we propose VideoDirectorGPT, a novel framework for
  consistent multi-scene video generation that uses the knowledge of LLMs for
  video content planning and grounded video generation. Specifically, given a
  single text prompt, we first ask our video planner LLM (GPT-4) to expand it
  into a 'video plan', which involves generating the scene descriptions, the
  entities with their respective layouts, the background for each scene, and
  consistency groupings of the entities and backgrounds. Next, guided by this
  output from the video planner, our video generator, Layout2Vid, has explicit
  control over spatial layouts and can maintain temporal consistency of
  entities/backgrounds across scenes, while only trained with image-level
  annotations. Our experiments demonstrate that VideoDirectorGPT framework
  substantially improves layout and movement control in both single- and
  multi-scene video generation and can generate multi-scene videos with visual
  consistency across scenes, while achieving competitive performance with SOTAs
  in open-domain single-scene T2V generation. We also demonstrate that our
  framework can dynamically control the strength for layout guidance and can also
  generate videos with user-provided images. We hope our framework can inspire
  future work on better integrating the planning ability of LLMs into consistent
  long video generation.
paper_authors._id: 651382954483b69098d5b406
paper_authors.name: Han Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-27T18:25:03.081Z
paper_authors._id: 646e86350867c99c2d3f2ecf
paper_authors.avatarUrl: /avatars/b89798ff623abffb169eacda2ac32fde.svg
paper_authors.fullname: Han Lin
paper_authors.type: user
paper_authors.user: hanlincs
paper_authors._id: 651382954483b69098d5b407
paper_authors.name: Abhay Zala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:21:03.479Z
paper_authors._id: 6361794819e7e4bf45741586
paper_authors.avatarUrl: /avatars/e4570b6ce752dc2b9b42969e21a50b57.svg
paper_authors.fullname: Abhay Zala
paper_authors.type: user
paper_authors.user: abhayzala
paper_authors._id: 651382954483b69098d5b408
paper_authors.name: Jaemin Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:21:27.282Z
paper_authors._id: 5ffe32d8942cf3533d364449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
paper_authors.fullname: Jaemin Cho
paper_authors.type: user
paper_authors.user: j-min
paper_authors._id: 651382954483b69098d5b409
paper_authors.name: Mohit Bansal
paper_title: VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided
    Planning
paper_summary: Although recent text-to-video (T2V) generation methods have seen significant
  advancements, most of these works focus on producing short video clips of a
  single event with a single background (i.e., single-scene videos). Meanwhile,
  recent large language models (LLMs) have demonstrated their capability in
  generating layouts and programs to control downstream visual modules such as
  image generation models. This raises an important question: can we leverage the
  knowledge embedded in these LLMs for temporally consistent long video
  generation? In this paper, we propose VideoDirectorGPT, a novel framework for
  consistent multi-scene video generation that uses the knowledge of LLMs for
  video content planning and grounded video generation. Specifically, given a
  single text prompt, we first ask our video planner LLM (GPT-4) to expand it
  into a 'video plan', which involves generating the scene descriptions, the
  entities with their respective layouts, the background for each scene, and
  consistency groupings of the entities and backgrounds. Next, guided by this
  output from the video planner, our video generator, Layout2Vid, has explicit
  control over spatial layouts and can maintain temporal consistency of
  entities/backgrounds across scenes, while only trained with image-level
  annotations. Our experiments demonstrate that VideoDirectorGPT framework
  substantially improves layout and movement control in both single- and
  multi-scene video generation and can generate multi-scene videos with visual
  consistency across scenes, while achieving competitive performance with SOTAs
  in open-domain single-scene T2V generation. We also demonstrate that our
  framework can dynamically control the strength for layout guidance and can also
  generate videos with user-provided images. We hope our framework can inspire
  future work on better integrating the planning ability of LLMs into consistent
  long video generation.
paper_ai_summary: VideoDirectorGPT combines LLMs for video content planning with a layout-controlled video generator, enabling temporally consistent multi-scene video generation from text prompts.
paper_ai_keywords: text-to-video (T2V) generation large language models (LLMs) video content planning grounded video generation video plan scene descriptions entities spacial layouts temporal consistency Layout2Vid open-domain single-scene T2V generation