Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach
<|reference_start|>Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach: Diffusion models have revolutionized image generation, and their extension to video generation has shown promise. However, current video diffusion models~(VDMs) rely on a scalar timestep variable applied at the clip level, which limits their ability to model complex temporal dependencies needed for various tasks like image-to-video generation. To address this limitation, we propose a frame-aware video diffusion model~(FVDM), which introduces a novel vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach allows each frame to follow an independent noise schedule, enhancing the model's capacity to capture fine-grained temporal dependencies. FVDM's flexibility is demonstrated across multiple tasks, including standard video generation, image-to-video generation, video interpolation, and long video synthesis. Through a diverse set of VTV configurations, we achieve superior quality in generated videos, overcoming challenges such as catastrophic forgetting during fine-tuning and limited generalizability in zero-shot methods.Our empirical evaluations show that FVDM outperforms state-of-the-art methods in video generation quality, while also excelling in extended tasks. By addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm in video synthesis, offering a robust framework with significant implications for generative modeling and multimedia applications.<|reference_end|>
@article{liu2024redefining,
    title={Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep
  Approach},
    author={Yaofang Liu, Yumeng Ren, Xiaodong Cun, Aitor Artola, Yang Liu, Tieyong
  Zeng, Raymond H. Chan, Jean-michel Morel},
    journal={arXiv preprint arXiv:2410.03160},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03160},
    primaryClass={cs.CV cs.LG}
}