title: VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video
    Generation
summary: Current video generation models excel at generating short clips but still
  struggle with creating multi-shot, movie-like videos. Existing models trained
  on large-scale data on the back of rich computational resources are
  unsurprisingly inadequate for maintaining a logical storyline and visual
  consistency across multiple shots of a cohesive script since they are often
  trained with a single-shot objective. To this end, we propose
  VideoGen-of-Thought (VGoT), a collaborative and training-free architecture
  designed specifically for multi-shot video generation. VGoT is designed with
  three goals in mind as follows. Multi-Shot Video Generation: We divide the
  video generation process into a structured, modular sequence, including (1)
  Script Generation, which translates a curt story into detailed prompts for each
  shot; (2) Keyframe Generation, responsible for creating visually consistent
  keyframes faithful to character portrayals; and (3) Shot-Level Video
  Generation, which transforms information from scripts and keyframes into shots;
  (4) Smoothing Mechanism that ensures a consistent multi-shot output. Reasonable
  Narrative Design: Inspired by cinematic scriptwriting, our prompt generation
  approach spans five key domains, ensuring logical consistency, character
  development, and narrative flow across the entire video. Cross-Shot
  Consistency: We ensure temporal and identity consistency by leveraging
  identity-preserving (IP) embeddings across shots, which are automatically
  created from the narrative. Additionally, we incorporate a cross-shot smoothing
  mechanism, which integrates a reset boundary that effectively combines latent
  features from adjacent shots, resulting in smooth transitions and maintaining
  visual coherence throughout the video. Our experiments demonstrate that VGoT
  surpasses existing video generation methods in producing high-quality,
  coherent, multi-shot videos.
paper_authors._id: 674fcf94d367eb55390fc796
paper_authors.name: Mingzhe Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:18:52.776Z
paper_authors._id: 62f0c4abe2999b231e5a893c
paper_authors.avatarUrl: /avatars/90da268b877a7ffe6665075c84018a83.svg
paper_authors.fullname: Mingzhe Zheng
paper_authors.type: user
paper_authors.user: Dunge0nMaster
paper_authors._id: 674fcf94d367eb55390fc797
paper_authors.name: Yongqi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:53:56.051Z
paper_authors._id: 6614caef2da465c162b52abc
paper_authors.avatarUrl: /avatars/81b71cb400c01ac10bfac9d1a4f2a9fc.svg
paper_authors.fullname: Yongqi Xu
paper_authors.type: user
paper_authors.user: Cheliosoops
paper_authors._id: 674fcf94d367eb55390fc798
paper_authors.name: Haojian Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:53:49.786Z
paper_authors._id: 657a776153e2fa36fed33259
paper_authors.avatarUrl: /avatars/f60777abe61bc6743e4cef0ede301295.svg
paper_authors.fullname: Haojian Huang
paper_authors.type: user
paper_authors.user: Jethro37
paper_authors._id: 674fcf94d367eb55390fc799
paper_authors.name: Xuran Ma
paper_authors._id: 674fcf94d367eb55390fc79a
paper_authors.name: Yexin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:53:28.015Z
paper_authors._id: 6497ff395b5d43c1c77d2a11
paper_authors.avatarUrl: /avatars/e3704701ed3ea0c1c165001521f40086.svg
paper_authors.fullname: Yexin Liu
paper_authors.type: user
paper_authors.user: AI4VR
paper_authors._id: 674fcf94d367eb55390fc79b
paper_authors.name: Wenjie Shu
paper_authors._id: 674fcf94d367eb55390fc79c
paper_authors.name: Yatian Pang
paper_authors._id: 674fcf94d367eb55390fc79d
paper_authors.name: Feilong Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:53:06.726Z
paper_authors._id: 67467d9bf7b7f6fffeee8f09
paper_authors.avatarUrl: /avatars/da5dd1acd79478540c32b34060710e1b.svg
paper_authors.fullname: tangfeilong
paper_authors.type: user
paper_authors.user: tfl01
paper_authors._id: 674fcf94d367eb55390fc79e
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:52:48.169Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 674fcf94d367eb55390fc79f
paper_authors.name: Harry Yang
paper_authors._id: 674fcf94d367eb55390fc7a0
paper_authors.name: Ser-Nam Lim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T10:36:17.200Z
paper_authors._id: 65884498ce38d143c435d279
paper_authors.avatarUrl: /avatars/96a917c40680a79a37dcb8fa75014c21.svg
paper_authors.fullname: Ser Nam Lim
paper_authors.type: user
paper_authors.user: sernam
paper_title: VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video
    Generation
paper_summary: Current video generation models excel at generating short clips but still
  struggle with creating multi-shot, movie-like videos. Existing models trained
  on large-scale data on the back of rich computational resources are
  unsurprisingly inadequate for maintaining a logical storyline and visual
  consistency across multiple shots of a cohesive script since they are often
  trained with a single-shot objective. To this end, we propose
  VideoGen-of-Thought (VGoT), a collaborative and training-free architecture
  designed specifically for multi-shot video generation. VGoT is designed with
  three goals in mind as follows. Multi-Shot Video Generation: We divide the
  video generation process into a structured, modular sequence, including (1)
  Script Generation, which translates a curt story into detailed prompts for each
  shot; (2) Keyframe Generation, responsible for creating visually consistent
  keyframes faithful to character portrayals; and (3) Shot-Level Video
  Generation, which transforms information from scripts and keyframes into shots;
  (4) Smoothing Mechanism that ensures a consistent multi-shot output. Reasonable
  Narrative Design: Inspired by cinematic scriptwriting, our prompt generation
  approach spans five key domains, ensuring logical consistency, character
  development, and narrative flow across the entire video. Cross-Shot
  Consistency: We ensure temporal and identity consistency by leveraging
  identity-preserving (IP) embeddings across shots, which are automatically
  created from the narrative. Additionally, we incorporate a cross-shot smoothing
  mechanism, which integrates a reset boundary that effectively combines latent
  features from adjacent shots, resulting in smooth transitions and maintaining
  visual coherence throughout the video. Our experiments demonstrate that VGoT
  surpasses existing video generation methods in producing high-quality,
  coherent, multi-shot videos.
paper_ai_summary: VGoT generates high-quality, coherent multi-shot videos by structuring the process into script, keyframe, and shot-level generation, using IP embeddings and smoothing mechanisms for consistency and narrative flow.
paper_ai_keywords: VideoGen-of-Thought VGoT multi-shot video generation script generation keyframe generation shot-level video generation smoothing mechanism identity-preserving embeddings IP embeddings cross-shot consistency