title: ControlVideo: Training-free Controllable Text-to-Video Generation
summary: Text-driven diffusion models have unlocked unprecedented abilities in image
  generation, whereas their video counterpart still lags behind due to the
  excessive training cost of temporal modeling. Besides the training burden, the
  generated videos also suffer from appearance inconsistency and structural
  flickers, especially in long video synthesis. To address these challenges, we
  design a training-free framework called ControlVideo to enable
  natural and efficient text-to-video generation. ControlVideo, adapted from
  ControlNet, leverages coarsely structural consistency from input motion
  sequences, and introduces three modules to improve video generation. Firstly,
  to ensure appearance coherence between frames, ControlVideo adds fully
  cross-frame interaction in self-attention modules. Secondly, to mitigate the
  flicker effect, it introduces an interleaved-frame smoother that employs frame
  interpolation on alternated frames. Finally, to produce long videos
  efficiently, it utilizes a hierarchical sampler that separately synthesizes
  each short clip with holistic coherency. Empowered with these modules,
  ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs
  quantitatively and qualitatively. Notably, thanks to the efficient designs, it
  generates both short and long videos within several minutes using one NVIDIA
  2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.
paper_authors._id: 646c140131968a60a022c3c4
paper_authors.name: Yabo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T07:51:28.713Z
paper_authors._id: 62cd752ba9be5c19555c2b4c
paper_authors.avatarUrl: /avatars/ed72684f0f2139516ccde24cd467cea6.svg
paper_authors.fullname: YaboZhang
paper_authors.type: user
paper_authors.user: Yabo
paper_authors._id: 646c140131968a60a022c3c5
paper_authors.name: Yuxiang Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T08:49:53.109Z
paper_authors._id: 6308bef17dc1b1a54cc9ff56
paper_authors.avatarUrl: /avatars/d3793f6192a7441a575e27996c065030.svg
paper_authors.fullname: Yuxiang Wei
paper_authors.type: user
paper_authors.user: csyxwei
paper_authors._id: 646c140131968a60a022c3c6
paper_authors.name: Dongsheng Jiang
paper_authors._id: 646c140131968a60a022c3c7
paper_authors.name: Xiaopeng Zhang
paper_authors._id: 646c140131968a60a022c3c8
paper_authors.name: Wangmeng Zuo
paper_authors._id: 646c140131968a60a022c3c9
paper_authors.name: Qi Tian
paper_title: ControlVideo: Training-free Controllable Text-to-Video Generation
paper_summary: Text-driven diffusion models have unlocked unprecedented abilities in image
  generation, whereas their video counterpart still lags behind due to the
  excessive training cost of temporal modeling. Besides the training burden, the
  generated videos also suffer from appearance inconsistency and structural
  flickers, especially in long video synthesis. To address these challenges, we
  design a training-free framework called ControlVideo to enable
  natural and efficient text-to-video generation. ControlVideo, adapted from
  ControlNet, leverages coarsely structural consistency from input motion
  sequences, and introduces three modules to improve video generation. Firstly,
  to ensure appearance coherence between frames, ControlVideo adds fully
  cross-frame interaction in self-attention modules. Secondly, to mitigate the
  flicker effect, it introduces an interleaved-frame smoother that employs frame
  interpolation on alternated frames. Finally, to produce long videos
  efficiently, it utilizes a hierarchical sampler that separately synthesizes
  each short clip with holistic coherency. Empowered with these modules,
  ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs
  quantitatively and qualitatively. Notably, thanks to the efficient designs, it
  generates both short and long videos within several minutes using one NVIDIA
  2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.
paper_ai_summary: ControlVideo, a training-free framework adapted from ControlNet, improves text-to-video generation by ensuring appearance coherence, reducing flicker, and efficiently producing long videos.
paper_ai_keywords: text-driven diffusion models video generation training-free framework ControlVideo ControlNet fully cross-frame interaction self-attention modules interleaved-frame smoother frame interpolation hierarchical sampler