title: VideoElevator: Elevating Video Generation Quality with Versatile
    Text-to-Image Diffusion Models
summary: Text-to-image diffusion models (T2I) have demonstrated unprecedented
  capabilities in creating realistic and aesthetic images. On the contrary,
  text-to-video diffusion models (T2V) still lag far behind in frame quality and
  text alignment, owing to insufficient quality and quantity of training videos.
  In this paper, we introduce VideoElevator, a training-free and plug-and-play
  method, which elevates the performance of T2V using superior capabilities of
  T2I. Different from conventional T2V sampling (i.e., temporal and spatial
  modeling), VideoElevator explicitly decomposes each sampling step into temporal
  motion refining and spatial quality elevating. Specifically, temporal motion
  refining uses encapsulated T2V to enhance temporal consistency, followed by
  inverting to the noise distribution required by T2I. Then, spatial quality
  elevating harnesses inflated T2I to directly predict less noisy latent, adding
  more photo-realistic details. We have conducted experiments in extensive
  prompts under the combination of various T2V and T2I. The results show that
  VideoElevator not only improves the performance of T2V baselines with
  foundational T2I, but also facilitates stylistic video synthesis with
  personalized T2I. Our code is available at
  https://github.com/YBYBZhang/VideoElevator.
paper_authors._id: 65ee742339f0e647aeb9fbd7
paper_authors.name: Yabo Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:52:14.675Z
paper_authors._id: 62cd752ba9be5c19555c2b4c
paper_authors.avatarUrl: /avatars/ed72684f0f2139516ccde24cd467cea6.svg
paper_authors.fullname: YaboZhang
paper_authors.type: user
paper_authors.user: Yabo
paper_authors._id: 65ee742339f0e647aeb9fbd8
paper_authors.name: Yuxiang Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:52:35.641Z
paper_authors._id: 6308bef17dc1b1a54cc9ff56
paper_authors.avatarUrl: /avatars/d3793f6192a7441a575e27996c065030.svg
paper_authors.fullname: Yuxiang Wei
paper_authors.type: user
paper_authors.user: csyxwei
paper_authors._id: 65ee742339f0e647aeb9fbd9
paper_authors.name: Xianhui Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:52:48.508Z
paper_authors._id: 6465e40a63e7e09dd02ef95d
paper_authors.avatarUrl: /avatars/2b7f2c55eeca13a0c45e9fe2bd31df36.svg
paper_authors.fullname: lin
paper_authors.type: user
paper_authors.user: xianhui
paper_authors._id: 65ee742339f0e647aeb9fbda
paper_authors.name: Zheng Hui
paper_authors._id: 65ee742339f0e647aeb9fbdb
paper_authors.name: Peiran Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:53:22.310Z
paper_authors._id: 64b74a45f902508f0d786505
paper_authors.avatarUrl: /avatars/8bc5aaa011642827e12524c4f0a56927.svg
paper_authors.fullname: Peiran REN
paper_authors.type: user
paper_authors.user: lyraestar
paper_authors._id: 65ee742339f0e647aeb9fbdc
paper_authors.name: Xuansong Xie
paper_authors._id: 65ee742339f0e647aeb9fbdd
paper_authors.name: Xiangyang Ji
paper_authors._id: 65ee742339f0e647aeb9fbde
paper_authors.name: Wangmeng Zuo
paper_title: VideoElevator: Elevating Video Generation Quality with Versatile
    Text-to-Image Diffusion Models
paper_summary: Text-to-image diffusion models (T2I) have demonstrated unprecedented
  capabilities in creating realistic and aesthetic images. On the contrary,
  text-to-video diffusion models (T2V) still lag far behind in frame quality and
  text alignment, owing to insufficient quality and quantity of training videos.
  In this paper, we introduce VideoElevator, a training-free and plug-and-play
  method, which elevates the performance of T2V using superior capabilities of
  T2I. Different from conventional T2V sampling (i.e., temporal and spatial
  modeling), VideoElevator explicitly decomposes each sampling step into temporal
  motion refining and spatial quality elevating. Specifically, temporal motion
  refining uses encapsulated T2V to enhance temporal consistency, followed by
  inverting to the noise distribution required by T2I. Then, spatial quality
  elevating harnesses inflated T2I to directly predict less noisy latent, adding
  more photo-realistic details. We have conducted experiments in extensive
  prompts under the combination of various T2V and T2I. The results show that
  VideoElevator not only improves the performance of T2V baselines with
  foundational T2I, but also facilitates stylistic video synthesis with
  personalized T2I. Our code is available at
  https://github.com/YBYBZhang/VideoElevator.
paper_ai_keywords: text-to-image diffusion models (T2I) text-to-video diffusion models (T2V) frame quality text alignment training videos VideoElevator training-free plug-and-play performance T2D MultiModN temporal motion refining spatial quality elevating temporal consistency noise distribution latent photo-realistic details experiments qualitative prompts foundational T2I stylistic video synthesis personalized T2I