title: VideoCrafter2: Overcoming Data Limitations for High-Quality Video
    Diffusion Models
summary: Text-to-video generation aims to produce a video based on a given prompt.
  Recently, several commercial video models have been able to generate plausible
  videos with minimal noise, excellent details, and high aesthetic scores.
  However, these models rely on large-scale, well-filtered, high-quality videos
  that are not accessible to the community. Many existing research works, which
  train models using the low-quality WebVid-10M dataset, struggle to generate
  high-quality videos because the models are optimized to fit WebVid-10M. In this
  work, we explore the training scheme of video models extended from Stable
  Diffusion and investigate the feasibility of leveraging low-quality videos and
  synthesized high-quality images to obtain a high-quality video model. We first
  analyze the connection between the spatial and temporal modules of video models
  and the distribution shift to low-quality videos. We observe that full training
  of all modules results in a stronger coupling between spatial and temporal
  modules than only training temporal modules. Based on this stronger coupling,
  we shift the distribution to higher quality without motion degradation by
  finetuning spatial modules with high-quality images, resulting in a generic
  high-quality video model. Evaluations are conducted to demonstrate the
  superiority of the proposed method, particularly in picture quality, motion,
  and concept composition.
paper_authors._id: 65a8938c2984fa7203d6cf93
paper_authors.name: Haoxin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:23:13.146Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 65a8938c2984fa7203d6cf94
paper_authors.name: Yong Zhang
paper_authors._id: 65a8938c2984fa7203d6cf95
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:10:17.994Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 65a8938c2984fa7203d6cf96
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:10:29.457Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 65a8938c2984fa7203d6cf97
paper_authors.name: Xintao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T08:19:38.538Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 65a8938c2984fa7203d6cf98
paper_authors.name: Chao Weng
paper_authors._id: 65a8938c2984fa7203d6cf99
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:11:09.585Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: VideoCrafter2: Overcoming Data Limitations for High-Quality Video
    Diffusion Models
paper_summary: Text-to-video generation aims to produce a video based on a given prompt.
  Recently, several commercial video models have been able to generate plausible
  videos with minimal noise, excellent details, and high aesthetic scores.
  However, these models rely on large-scale, well-filtered, high-quality videos
  that are not accessible to the community. Many existing research works, which
  train models using the low-quality WebVid-10M dataset, struggle to generate
  high-quality videos because the models are optimized to fit WebVid-10M. In this
  work, we explore the training scheme of video models extended from Stable
  Diffusion and investigate the feasibility of leveraging low-quality videos and
  synthesized high-quality images to obtain a high-quality video model. We first
  analyze the connection between the spatial and temporal modules of video models
  and the distribution shift to low-quality videos. We observe that full training
  of all modules results in a stronger coupling between spatial and temporal
  modules than only training temporal modules. Based on this stronger coupling,
  we shift the distribution to higher quality without motion degradation by
  finetuning spatial modules with high-quality images, resulting in a generic
  high-quality video model. Evaluations are conducted to demonstrate the
  superiority of the proposed method, particularly in picture quality, motion,
  and concept composition.
paper_ai_keywords: Stable Diffusion distribution shift spatial modules temporal modules full training finetuning