Fine-gained Zero-shot Video Sampling
Incorporating a temporal dimension into pretrained image diffusion models for
video generation is a prevalent approach. However, this method is
computationally demanding and necessitates large-scale video datasets. More
critically, the heterogeneity between image and video datasets often results in
catastrophic forgetting of the image expertise. Recent attempts to directly
extract video snippets from image diffusion models have somewhat mitigated
these problems. Nevertheless, these methods can only generate brief video clips
with simple movements and fail to capture fine-grained motion or non-grid
deformation. In this paper, we propose a novel Zero-Shot video Sampling
algorithm, denoted as ZS^2, capable of directly sampling
high-quality video clips from existing image synthesis methods, such as Stable
Diffusion, without any training or optimization. Specifically, ZS^2
utilizes the dependency noise model and temporal momentum attention to ensure
content consistency and animation coherence, respectively. This ability enables
it to excel in related tasks, such as conditional and context-specialized video
generation and instruction-guided video editing. Experimental results
demonstrate that ZS^2 achieves state-of-the-art performance in
zero-shot video generation, occasionally outperforming recent supervised
methods.
  Homepage: https://densechen.github.io/zss/.
66aae82fbe57064af170e845
Dengsheng Chen
66aae82fbe57064af170e846
Jie Hu
66aae82fbe57064af170e847
Xiaoming Wei
66aae82fbe57064af170e848
Enhua Wu
Fine-gained Zero-shot Video Sampling
Incorporating a temporal dimension into pretrained image diffusion models for
video generation is a prevalent approach. However, this method is
computationally demanding and necessitates large-scale video datasets. More
critically, the heterogeneity between image and video datasets often results in
catastrophic forgetting of the image expertise. Recent attempts to directly
extract video snippets from image diffusion models have somewhat mitigated
these problems. Nevertheless, these methods can only generate brief video clips
with simple movements and fail to capture fine-grained motion or non-grid
deformation. In this paper, we propose a novel Zero-Shot video Sampling
algorithm, denoted as ZS^2, capable of directly sampling
high-quality video clips from existing image synthesis methods, such as Stable
Diffusion, without any training or optimization. Specifically, ZS^2
utilizes the dependency noise model and temporal momentum attention to ensure
content consistency and animation coherence, respectively. This ability enables
it to excel in related tasks, such as conditional and context-specialized video
generation and instruction-guided video editing. Experimental results
demonstrate that ZS^2 achieves state-of-the-art performance in
zero-shot video generation, occasionally outperforming recent supervised
methods.
  Homepage: https://densechen.github.io/zss/.
A Zero-Shot video Sampling algorithm, $\mathcal{ZS}^2$, generates high-quality video clips using image synthesis models by leveraging dependency noise models and temporal momentum attention, achieving state-of-the-art performance in video generation.
image diffusion models video generation dependency noise model temporal momentum attention conditional video generation context-specialized video generation instruction-guided video editing video snippets