Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep
  Approach
Diffusion models have revolutionized image generation, and their extension to
video generation has shown promise. However, current video diffusion
models~(VDMs) rely on a scalar timestep variable applied at the clip level,
which limits their ability to model complex temporal dependencies needed for
various tasks like image-to-video generation. To address this limitation, we
propose a frame-aware video diffusion model~(FVDM), which introduces a novel
vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach
allows each frame to follow an independent noise schedule, enhancing the
model's capacity to capture fine-grained temporal dependencies. FVDM's
flexibility is demonstrated across multiple tasks, including standard video
generation, image-to-video generation, video interpolation, and long video
synthesis. Through a diverse set of VTV configurations, we achieve superior
quality in generated videos, overcoming challenges such as catastrophic
forgetting during fine-tuning and limited generalizability in zero-shot
methods.Our empirical evaluations show that FVDM outperforms state-of-the-art
methods in video generation quality, while also excelling in extended tasks. By
addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm
in video synthesis, offering a robust framework with significant implications
for generative modeling and multimedia applications.
6705853cc16d934c94287f63
Yaofang Liu
claimed_verified
2024-10-09T07:37:31.501Z
658d46649a1397992a4d657f
/avatars/66d23095b8d0368ea862778dbf123a67.svg
Raphael Liu
user
RaphaelLiu
6705853cc16d934c94287f64
Yumeng Ren
6705853cc16d934c94287f65
Xiaodong Cun
claimed_verified
2025-06-30T06:23:58.558Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
6705853cc16d934c94287f66
Aitor Artola
6705853cc16d934c94287f67
Yang Liu
6705853cc16d934c94287f68
Tieyong Zeng
6705853cc16d934c94287f69
Raymond H. Chan
6705853cc16d934c94287f6a
Jean-michel Morel
Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep
  Approach
Diffusion models have revolutionized image generation, and their extension to
video generation has shown promise. However, current video diffusion
models~(VDMs) rely on a scalar timestep variable applied at the clip level,
which limits their ability to model complex temporal dependencies needed for
various tasks like image-to-video generation. To address this limitation, we
propose a frame-aware video diffusion model~(FVDM), which introduces a novel
vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach
allows each frame to follow an independent noise schedule, enhancing the
model's capacity to capture fine-grained temporal dependencies. FVDM's
flexibility is demonstrated across multiple tasks, including standard video
generation, image-to-video generation, video interpolation, and long video
synthesis. Through a diverse set of VTV configurations, we achieve superior
quality in generated videos, overcoming challenges such as catastrophic
forgetting during fine-tuning and limited generalizability in zero-shot
methods.Our empirical evaluations show that FVDM outperforms state-of-the-art
methods in video generation quality, while also excelling in extended tasks. By
addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm
in video synthesis, offering a robust framework with significant implications
for generative modeling and multimedia applications.
Frame-aware video diffusion model enhances temporal dependencies and quality in video generation tasks by using a vectorized timestep variable.
diffusion models video generation scalar timestep frame-aware video diffusion model vectorized timestep variable noise schedule fine-grained temporal dependencies catastrophic forgetting zero-shot methods standard video generation image-to-video generation video interpolation long video synthesis