ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video
  Generation
Video generation has made remarkable progress in recent years, especially
since the advent of the video diffusion models. Many video generation models
can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD).
However, most video models can only generate low frame rate videos due to the
limited GPU memory as well as the difficulty of modeling a large set of frames.
The training videos are always uniformly sampled at a specified interval for
temporal compression. Previous methods promote the frame rate by either
training a video interpolation model in pixel space as a postprocessing stage
or training an interpolation model in latent space for a specific base video
model. In this paper, we propose a training-free video interpolation method for
generative video diffusion models, which is generalizable to different models
in a plug-and-play manner. We investigate the non-linearity in the feature
space of video diffusion models and transform a video model into a
self-cascaded video diffusion model with incorporating the designed hidden
state correction modules. The self-cascaded architecture and the correction
module are proposed to retain the temporal consistency between key frames and
the interpolated frames. Extensive evaluations are preformed on multiple
popular video models to demonstrate the effectiveness of the propose method,
especially that our training-free method is even comparable to trained
interpolation models supported by huge compute resources and large-scale
datasets.
665e91edaed2cd03b6509663
Shaoshu Yang
665e91edaed2cd03b6509664
Yong Zhang
claimed_verified
2024-06-04T13:12:57.754Z
6311b3f3fdb55de45d1fd2bd
/avatars/855597bd5814d2aa7818da7e70ea2694.svg
F
user
norris
665e91edaed2cd03b6509665
Xiaodong Cun
admin_assigned
2024-06-04T08:27:58.678Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
665e91edaed2cd03b6509666
Ying Shan
665e91edaed2cd03b6509667
Ran He
ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video
  Generation
Video generation has made remarkable progress in recent years, especially
since the advent of the video diffusion models. Many video generation models
can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD).
However, most video models can only generate low frame rate videos due to the
limited GPU memory as well as the difficulty of modeling a large set of frames.
The training videos are always uniformly sampled at a specified interval for
temporal compression. Previous methods promote the frame rate by either
training a video interpolation model in pixel space as a postprocessing stage
or training an interpolation model in latent space for a specific base video
model. In this paper, we propose a training-free video interpolation method for
generative video diffusion models, which is generalizable to different models
in a plug-and-play manner. We investigate the non-linearity in the feature
space of video diffusion models and transform a video model into a
self-cascaded video diffusion model with incorporating the designed hidden
state correction modules. The self-cascaded architecture and the correction
module are proposed to retain the temporal consistency between key frames and
the interpolated frames. Extensive evaluations are preformed on multiple
popular video models to demonstrate the effectiveness of the propose method,
especially that our training-free method is even comparable to trained
interpolation models supported by huge compute resources and large-scale
datasets.
A training-free video interpolation method enhances frame rate in generative diffusion models by incorporating hidden state correction modules within a self-cascaded architecture, achieving performance comparable to models trained with extensive computational resources.
video diffusion models Stable Video Diffusion (SVD) frame rate GPU memory latent space training-free video interpolation self-cascaded architecture hidden state correction modules temporal consistency