title: VEnhancer: Generative Space-Time Enhancement for Video Generation
summary: We present VEnhancer, a generative space-time enhancement framework that
  improves the existing text-to-video results by adding more details in spatial
  domain and synthetic detailed motion in temporal domain. Given a generated
  low-quality video, our approach can increase its spatial and temporal
  resolution simultaneously with arbitrary up-sampling space and time scales
  through a unified video diffusion model. Furthermore, VEnhancer effectively
  removes generated spatial artifacts and temporal flickering of generated
  videos. To achieve this, basing on a pretrained video diffusion model, we train
  a video ControlNet and inject it to the diffusion model as a condition on low
  frame-rate and low-resolution videos. To effectively train this video
  ControlNet, we design space-time data augmentation as well as video-aware
  conditioning. Benefiting from the above designs, VEnhancer yields to be stable
  during training and shares an elegant end-to-end training manner. Extensive
  experiments show that VEnhancer surpasses existing state-of-the-art video
  super-resolution and space-time super-resolution methods in enhancing
  AI-generated videos. Moreover, with VEnhancer, exisiting open-source
  state-of-the-art text-to-video method, VideoCrafter-2, reaches the top one in
  video generation benchmark -- VBench.
paper_authors._id: 668f46fdd3b02463a73bc21c
paper_authors.name: Jingwen He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T15:01:38.918Z
paper_authors._id: 66a49f9b2b460286b05646b8
paper_authors.avatarUrl: /avatars/5123cf7c3d00b5bc47a86011afd6abe8.svg
paper_authors.fullname: jwhe
paper_authors.type: user
paper_authors.user: jwhejwhe
paper_authors._id: 668f46fdd3b02463a73bc21d
paper_authors.name: Tianfan Xue
paper_authors._id: 668f46fdd3b02463a73bc21e
paper_authors.name: Dongyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T15:24:51.297Z
paper_authors._id: 646f1bef075e11ca78da3bb7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
paper_authors.fullname: Dongyang Liu (Chris Liu)
paper_authors.type: user
paper_authors.user: Cxxs
paper_authors._id: 668f46fdd3b02463a73bc21f
paper_authors.name: Xinqi Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:35:02.938Z
paper_authors._id: 644b221ef434a6a63b1a8f04
paper_authors.avatarUrl: /avatars/34d106f0fd536cc7fbeb3abb94b5d775.svg
paper_authors.fullname: qilin
paper_authors.type: user
paper_authors.user: xinqilin
paper_authors._id: 668f46fdd3b02463a73bc220
paper_authors.name: Peng Gao
paper_authors._id: 668f46fdd3b02463a73bc221
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:35:28.492Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 668f46fdd3b02463a73bc222
paper_authors.name: Yu Qiao
paper_authors._id: 668f46fdd3b02463a73bc223
paper_authors.name: Wanli Ouyang
paper_authors._id: 668f46fdd3b02463a73bc224
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:34:28.342Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: VEnhancer: Generative Space-Time Enhancement for Video Generation
paper_summary: We present VEnhancer, a generative space-time enhancement framework that
  improves the existing text-to-video results by adding more details in spatial
  domain and synthetic detailed motion in temporal domain. Given a generated
  low-quality video, our approach can increase its spatial and temporal
  resolution simultaneously with arbitrary up-sampling space and time scales
  through a unified video diffusion model. Furthermore, VEnhancer effectively
  removes generated spatial artifacts and temporal flickering of generated
  videos. To achieve this, basing on a pretrained video diffusion model, we train
  a video ControlNet and inject it to the diffusion model as a condition on low
  frame-rate and low-resolution videos. To effectively train this video
  ControlNet, we design space-time data augmentation as well as video-aware
  conditioning. Benefiting from the above designs, VEnhancer yields to be stable
  during training and shares an elegant end-to-end training manner. Extensive
  experiments show that VEnhancer surpasses existing state-of-the-art video
  super-resolution and space-time super-resolution methods in enhancing
  AI-generated videos. Moreover, with VEnhancer, exisiting open-source
  state-of-the-art text-to-video method, VideoCrafter-2, reaches the top one in
  video generation benchmark -- VBench.
paper_ai_keywords: gener terms specific to deep learning in this context: generative space-time enhancement unified video diffusion model video ControlNet space-time data augmentation video-aware conditioning end-to-end training video super-resolution space-time super-resolution AI-generated videos benchmark VBench