title: ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video
    Generation
summary: Video generation has made remarkable progress in recent years, especially
  since the advent of the video diffusion models. Many video generation models
  can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD).
  However, most video models can only generate low frame rate videos due to the
  limited GPU memory as well as the difficulty of modeling a large set of frames.
  The training videos are always uniformly sampled at a specified interval for
  temporal compression. Previous methods promote the frame rate by either
  training a video interpolation model in pixel space as a postprocessing stage
  or training an interpolation model in latent space for a specific base video
  model. In this paper, we propose a training-free video interpolation method for
  generative video diffusion models, which is generalizable to different models
  in a plug-and-play manner. We investigate the non-linearity in the feature
  space of video diffusion models and transform a video model into a
  self-cascaded video diffusion model with incorporating the designed hidden
  state correction modules. The self-cascaded architecture and the correction
  module are proposed to retain the temporal consistency between key frames and
  the interpolated frames. Extensive evaluations are preformed on multiple
  popular video models to demonstrate the effectiveness of the propose method,
  especially that our training-free method is even comparable to trained
  interpolation models supported by huge compute resources and large-scale
  datasets.
paper_authors._id: 665e91edaed2cd03b6509663
paper_authors.name: Shaoshu Yang
paper_authors._id: 665e91edaed2cd03b6509664
paper_authors.name: Yong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-04T13:12:57.754Z
paper_authors._id: 6311b3f3fdb55de45d1fd2bd
paper_authors.avatarUrl: /avatars/855597bd5814d2aa7818da7e70ea2694.svg
paper_authors.fullname: F
paper_authors.type: user
paper_authors.user: norris
paper_authors._id: 665e91edaed2cd03b6509665
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:27:58.678Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 665e91edaed2cd03b6509666
paper_authors.name: Ying Shan
paper_authors._id: 665e91edaed2cd03b6509667
paper_authors.name: Ran He
paper_title: ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video
    Generation
paper_summary: Video generation has made remarkable progress in recent years, especially
  since the advent of the video diffusion models. Many video generation models
  can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD).
  However, most video models can only generate low frame rate videos due to the
  limited GPU memory as well as the difficulty of modeling a large set of frames.
  The training videos are always uniformly sampled at a specified interval for
  temporal compression. Previous methods promote the frame rate by either
  training a video interpolation model in pixel space as a postprocessing stage
  or training an interpolation model in latent space for a specific base video
  model. In this paper, we propose a training-free video interpolation method for
  generative video diffusion models, which is generalizable to different models
  in a plug-and-play manner. We investigate the non-linearity in the feature
  space of video diffusion models and transform a video model into a
  self-cascaded video diffusion model with incorporating the designed hidden
  state correction modules. The self-cascaded architecture and the correction
  module are proposed to retain the temporal consistency between key frames and
  the interpolated frames. Extensive evaluations are preformed on multiple
  popular video models to demonstrate the effectiveness of the propose method,
  especially that our training-free method is even comparable to trained
  interpolation models supported by huge compute resources and large-scale
  datasets.
paper_ai_keywords: video diffusion models Stable Video Diffusion (SVD) pixel space latent space video interpolation model self-cascaded video diffusion model hidden state correction modules temporal consistency feature space