title: VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM
summary: The recent innovations and breakthroughs in diffusion models have
  significantly expanded the possibilities of generating high-quality videos for
  the given prompts. Most existing works tackle the single-scene scenario with
  only one video event occurring in a single background. Extending to generate
  multi-scene videos nevertheless is not trivial and necessitates to nicely
  manage the logic in between while preserving the consistent visual appearance
  of key content across video scenes. In this paper, we propose a novel
  framework, namely VideoDrafter, for content-consistent multi-scene video
  generation. Technically, VideoDrafter leverages Large Language Models (LLM) to
  convert the input prompt into comprehensive multi-scene script that benefits
  from the logical knowledge learnt by LLM. The script for each scene includes a
  prompt describing the event, the foreground/background entities, as well as
  camera movement. VideoDrafter identifies the common entities throughout the
  script and asks LLM to detail each entity. The resultant entity description is
  then fed into a text-to-image model to generate a reference image for each
  entity. Finally, VideoDrafter outputs a multi-scene video by generating each
  scene video via a diffusion process that takes the reference images, the
  descriptive prompt of the event and camera movement into account. The diffusion
  model incorporates the reference images as the condition and alignment to
  strengthen the content consistency of multi-scene videos. Extensive experiments
  demonstrate that VideoDrafter outperforms the SOTA video generation models in
  terms of visual quality, content consistency, and user preference.
paper_authors._id: 6594bf6b310642fd01448b73
paper_authors.name: Fuchen Long
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T09:41:33.446Z
paper_authors._id: 6449f2dfeb7db8f70fb990f8
paper_authors.avatarUrl: /avatars/72c02e754bb35dab05c4a5f1e69c95f1.svg
paper_authors.fullname: Fuchen
paper_authors.type: user
paper_authors.user: FireCRT
paper_authors._id: 6594bf6b310642fd01448b74
paper_authors.name: Zhaofan Qiu
paper_authors._id: 6594bf6b310642fd01448b75
paper_authors.name: Ting Yao
paper_authors._id: 6594bf6b310642fd01448b76
paper_authors.name: Tao Mei
paper_title: VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM
paper_summary: The recent innovations and breakthroughs in diffusion models have
  significantly expanded the possibilities of generating high-quality videos for
  the given prompts. Most existing works tackle the single-scene scenario with
  only one video event occurring in a single background. Extending to generate
  multi-scene videos nevertheless is not trivial and necessitates to nicely
  manage the logic in between while preserving the consistent visual appearance
  of key content across video scenes. In this paper, we propose a novel
  framework, namely VideoDrafter, for content-consistent multi-scene video
  generation. Technically, VideoDrafter leverages Large Language Models (LLM) to
  convert the input prompt into comprehensive multi-scene script that benefits
  from the logical knowledge learnt by LLM. The script for each scene includes a
  prompt describing the event, the foreground/background entities, as well as
  camera movement. VideoDrafter identifies the common entities throughout the
  script and asks LLM to detail each entity. The resultant entity description is
  then fed into a text-to-image model to generate a reference image for each
  entity. Finally, VideoDrafter outputs a multi-scene video by generating each
  scene video via a diffusion process that takes the reference images, the
  descriptive prompt of the event and camera movement into account. The diffusion
  model incorporates the reference images as the condition and alignment to
  strengthen the content consistency of multi-scene videos. Extensive experiments
  demonstrate that VideoDrafter outperforms the SOTA video generation models in
  terms of visual quality, content consistency, and user preference.
paper_ai_summary: VideoDrafter generates high-quality, content-consistent multi-scene videos using Large Language Models and diffusion models, enhancing visual quality and user preference by leveraging comprehensive scripts and reference images.
paper_ai_keywords: diffusion models Large Language Models (LLM) multi-scene video generation text-to-image model reference images content consistency visual quality user preference camera movement