VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM
The recent innovations and breakthroughs in diffusion models have
significantly expanded the possibilities of generating high-quality videos for
the given prompts. Most existing works tackle the single-scene scenario with
only one video event occurring in a single background. Extending to generate
multi-scene videos nevertheless is not trivial and necessitates to nicely
manage the logic in between while preserving the consistent visual appearance
of key content across video scenes. In this paper, we propose a novel
framework, namely VideoDrafter, for content-consistent multi-scene video
generation. Technically, VideoDrafter leverages Large Language Models (LLM) to
convert the input prompt into comprehensive multi-scene script that benefits
from the logical knowledge learnt by LLM. The script for each scene includes a
prompt describing the event, the foreground/background entities, as well as
camera movement. VideoDrafter identifies the common entities throughout the
script and asks LLM to detail each entity. The resultant entity description is
then fed into a text-to-image model to generate a reference image for each
entity. Finally, VideoDrafter outputs a multi-scene video by generating each
scene video via a diffusion process that takes the reference images, the
descriptive prompt of the event and camera movement into account. The diffusion
model incorporates the reference images as the condition and alignment to
strengthen the content consistency of multi-scene videos. Extensive experiments
demonstrate that VideoDrafter outperforms the SOTA video generation models in
terms of visual quality, content consistency, and user preference.
6594bf6b310642fd01448b73
Fuchen Long
claimed_verified
2025-01-16T09:41:33.446Z
6449f2dfeb7db8f70fb990f8
/avatars/72c02e754bb35dab05c4a5f1e69c95f1.svg
Fuchen
user
FireCRT
6594bf6b310642fd01448b74
Zhaofan Qiu
6594bf6b310642fd01448b75
Ting Yao
6594bf6b310642fd01448b76
Tao Mei
VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM
The recent innovations and breakthroughs in diffusion models have
significantly expanded the possibilities of generating high-quality videos for
the given prompts. Most existing works tackle the single-scene scenario with
only one video event occurring in a single background. Extending to generate
multi-scene videos nevertheless is not trivial and necessitates to nicely
manage the logic in between while preserving the consistent visual appearance
of key content across video scenes. In this paper, we propose a novel
framework, namely VideoDrafter, for content-consistent multi-scene video
generation. Technically, VideoDrafter leverages Large Language Models (LLM) to
convert the input prompt into comprehensive multi-scene script that benefits
from the logical knowledge learnt by LLM. The script for each scene includes a
prompt describing the event, the foreground/background entities, as well as
camera movement. VideoDrafter identifies the common entities throughout the
script and asks LLM to detail each entity. The resultant entity description is
then fed into a text-to-image model to generate a reference image for each
entity. Finally, VideoDrafter outputs a multi-scene video by generating each
scene video via a diffusion process that takes the reference images, the
descriptive prompt of the event and camera movement into account. The diffusion
model incorporates the reference images as the condition and alignment to
strengthen the content consistency of multi-scene videos. Extensive experiments
demonstrate that VideoDrafter outperforms the SOTA video generation models in
terms of visual quality, content consistency, and user preference.
VideoDrafter generates high-quality, content-consistent multi-scene videos using Large Language Models and diffusion models, enhancing visual quality and user preference by leveraging comprehensive scripts and reference images.
diffusion models Large Language Models (LLM) multi-scene video generation text-to-image model reference images content consistency visual quality user preference camera movement