VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided
  Planning
Although recent text-to-video (T2V) generation methods have seen significant
advancements, most of these works focus on producing short video clips of a
single event with a single background (i.e., single-scene videos). Meanwhile,
recent large language models (LLMs) have demonstrated their capability in
generating layouts and programs to control downstream visual modules such as
image generation models. This raises an important question: can we leverage the
knowledge embedded in these LLMs for temporally consistent long video
generation? In this paper, we propose VideoDirectorGPT, a novel framework for
consistent multi-scene video generation that uses the knowledge of LLMs for
video content planning and grounded video generation. Specifically, given a
single text prompt, we first ask our video planner LLM (GPT-4) to expand it
into a 'video plan', which involves generating the scene descriptions, the
entities with their respective layouts, the background for each scene, and
consistency groupings of the entities and backgrounds. Next, guided by this
output from the video planner, our video generator, Layout2Vid, has explicit
control over spatial layouts and can maintain temporal consistency of
entities/backgrounds across scenes, while only trained with image-level
annotations. Our experiments demonstrate that VideoDirectorGPT framework
substantially improves layout and movement control in both single- and
multi-scene video generation and can generate multi-scene videos with visual
consistency across scenes, while achieving competitive performance with SOTAs
in open-domain single-scene T2V generation. We also demonstrate that our
framework can dynamically control the strength for layout guidance and can also
generate videos with user-provided images. We hope our framework can inspire
future work on better integrating the planning ability of LLMs into consistent
long video generation.
651382954483b69098d5b406
Han Lin
claimed_verified
2023-09-27T18:25:03.081Z
646e86350867c99c2d3f2ecf
/avatars/b89798ff623abffb169eacda2ac32fde.svg
Han Lin
user
hanlincs
651382954483b69098d5b407
Abhay Zala
admin_assigned
2023-09-27T10:21:03.479Z
6361794819e7e4bf45741586
/avatars/e4570b6ce752dc2b9b42969e21a50b57.svg
Abhay Zala
user
abhayzala
651382954483b69098d5b408
Jaemin Cho
admin_assigned
2023-09-27T10:21:27.282Z
5ffe32d8942cf3533d364449
https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
Jaemin Cho
user
j-min
651382954483b69098d5b409
Mohit Bansal
VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided
  Planning
Although recent text-to-video (T2V) generation methods have seen significant
advancements, most of these works focus on producing short video clips of a
single event with a single background (i.e., single-scene videos). Meanwhile,
recent large language models (LLMs) have demonstrated their capability in
generating layouts and programs to control downstream visual modules such as
image generation models. This raises an important question: can we leverage the
knowledge embedded in these LLMs for temporally consistent long video
generation? In this paper, we propose VideoDirectorGPT, a novel framework for
consistent multi-scene video generation that uses the knowledge of LLMs for
video content planning and grounded video generation. Specifically, given a
single text prompt, we first ask our video planner LLM (GPT-4) to expand it
into a 'video plan', which involves generating the scene descriptions, the
entities with their respective layouts, the background for each scene, and
consistency groupings of the entities and backgrounds. Next, guided by this
output from the video planner, our video generator, Layout2Vid, has explicit
control over spatial layouts and can maintain temporal consistency of
entities/backgrounds across scenes, while only trained with image-level
annotations. Our experiments demonstrate that VideoDirectorGPT framework
substantially improves layout and movement control in both single- and
multi-scene video generation and can generate multi-scene videos with visual
consistency across scenes, while achieving competitive performance with SOTAs
in open-domain single-scene T2V generation. We also demonstrate that our
framework can dynamically control the strength for layout guidance and can also
generate videos with user-provided images. We hope our framework can inspire
future work on better integrating the planning ability of LLMs into consistent
long video generation.
VideoDirectorGPT combines LLMs for video content planning with a layout-controlled video generator, enabling temporally consistent multi-scene video generation from text prompts.
text-to-video (T2V) generation large language models (LLMs) video content planning grounded video generation video plan scene descriptions entities spacial layouts temporal consistency Layout2Vid open-domain single-scene T2V generation