title: Fine-gained Zero-shot Video Sampling
summary: Incorporating a temporal dimension into pretrained image diffusion models for
  video generation is a prevalent approach. However, this method is
  computationally demanding and necessitates large-scale video datasets. More
  critically, the heterogeneity between image and video datasets often results in
  catastrophic forgetting of the image expertise. Recent attempts to directly
  extract video snippets from image diffusion models have somewhat mitigated
  these problems. Nevertheless, these methods can only generate brief video clips
  with simple movements and fail to capture fine-grained motion or non-grid
  deformation. In this paper, we propose a novel Zero-Shot video Sampling
  algorithm, denoted as ZS^2, capable of directly sampling
  high-quality video clips from existing image synthesis methods, such as Stable
  Diffusion, without any training or optimization. Specifically, ZS^2
  utilizes the dependency noise model and temporal momentum attention to ensure
  content consistency and animation coherence, respectively. This ability enables
  it to excel in related tasks, such as conditional and context-specialized video
  generation and instruction-guided video editing. Experimental results
  demonstrate that ZS^2 achieves state-of-the-art performance in
  zero-shot video generation, occasionally outperforming recent supervised
  methods.
    Homepage: https://densechen.github.io/zss/.
paper_authors._id: 66aae82fbe57064af170e845
paper_authors.name: Dengsheng Chen
paper_authors._id: 66aae82fbe57064af170e846
paper_authors.name: Jie Hu
paper_authors._id: 66aae82fbe57064af170e847
paper_authors.name: Xiaoming Wei
paper_authors._id: 66aae82fbe57064af170e848
paper_authors.name: Enhua Wu
paper_title: Fine-gained Zero-shot Video Sampling
paper_summary: Incorporating a temporal dimension into pretrained image diffusion models for
  video generation is a prevalent approach. However, this method is
  computationally demanding and necessitates large-scale video datasets. More
  critically, the heterogeneity between image and video datasets often results in
  catastrophic forgetting of the image expertise. Recent attempts to directly
  extract video snippets from image diffusion models have somewhat mitigated
  these problems. Nevertheless, these methods can only generate brief video clips
  with simple movements and fail to capture fine-grained motion or non-grid
  deformation. In this paper, we propose a novel Zero-Shot video Sampling
  algorithm, denoted as ZS^2, capable of directly sampling
  high-quality video clips from existing image synthesis methods, such as Stable
  Diffusion, without any training or optimization. Specifically, ZS^2
  utilizes the dependency noise model and temporal momentum attention to ensure
  content consistency and animation coherence, respectively. This ability enables
  it to excel in related tasks, such as conditional and context-specialized video
  generation and instruction-guided video editing. Experimental results
  demonstrate that ZS^2 achieves state-of-the-art performance in
  zero-shot video generation, occasionally outperforming recent supervised
  methods.
    Homepage: https://densechen.github.io/zss/.
paper_ai_keywords: diffusion models video generation computational demanding large-scale video datasets catastrophic forgetting image expertise video snippets temporal dimension high-quality video clips simple movements fine-grained motion non-grid deformation Zero-Shot video Sampling $\mathcal{ZS}^2$ Stable Diffusion dependency noise model temporal momentum attention content consistency animation coherence conditional video generation context-specialized video generation instruction-guided video editing zero-shot video generation supervised methods