title: Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep
    Approach
summary: Diffusion models have revolutionized image generation, and their extension to
  video generation has shown promise. However, current video diffusion
  models~(VDMs) rely on a scalar timestep variable applied at the clip level,
  which limits their ability to model complex temporal dependencies needed for
  various tasks like image-to-video generation. To address this limitation, we
  propose a frame-aware video diffusion model~(FVDM), which introduces a novel
  vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach
  allows each frame to follow an independent noise schedule, enhancing the
  model's capacity to capture fine-grained temporal dependencies. FVDM's
  flexibility is demonstrated across multiple tasks, including standard video
  generation, image-to-video generation, video interpolation, and long video
  synthesis. Through a diverse set of VTV configurations, we achieve superior
  quality in generated videos, overcoming challenges such as catastrophic
  forgetting during fine-tuning and limited generalizability in zero-shot
  methods.Our empirical evaluations show that FVDM outperforms state-of-the-art
  methods in video generation quality, while also excelling in extended tasks. By
  addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm
  in video synthesis, offering a robust framework with significant implications
  for generative modeling and multimedia applications.
paper_authors._id: 6705853cc16d934c94287f63
paper_authors.name: Yaofang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T07:37:31.501Z
paper_authors._id: 658d46649a1397992a4d657f
paper_authors.avatarUrl: /avatars/66d23095b8d0368ea862778dbf123a67.svg
paper_authors.fullname: Raphael Liu
paper_authors.type: user
paper_authors.user: RaphaelLiu
paper_authors._id: 6705853cc16d934c94287f64
paper_authors.name: Yumeng Ren
paper_authors._id: 6705853cc16d934c94287f65
paper_authors.name: Xiaodong Cun
paper_authors._id: 6705853cc16d934c94287f66
paper_authors.name: Aitor Artola
paper_authors._id: 6705853cc16d934c94287f67
paper_authors.name: Yang Liu
paper_authors._id: 6705853cc16d934c94287f68
paper_authors.name: Tieyong Zeng
paper_authors._id: 6705853cc16d934c94287f69
paper_authors.name: Raymond H. Chan
paper_authors._id: 6705853cc16d934c94287f6a
paper_authors.name: Jean-michel Morel
paper_title: Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep
    Approach
paper_summary: Diffusion models have revolutionized image generation, and their extension to
  video generation has shown promise. However, current video diffusion
  models~(VDMs) rely on a scalar timestep variable applied at the clip level,
  which limits their ability to model complex temporal dependencies needed for
  various tasks like image-to-video generation. To address this limitation, we
  propose a frame-aware video diffusion model~(FVDM), which introduces a novel
  vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach
  allows each frame to follow an independent noise schedule, enhancing the
  model's capacity to capture fine-grained temporal dependencies. FVDM's
  flexibility is demonstrated across multiple tasks, including standard video
  generation, image-to-video generation, video interpolation, and long video
  synthesis. Through a diverse set of VTV configurations, we achieve superior
  quality in generated videos, overcoming challenges such as catastrophic
  forgetting during fine-tuning and limited generalizability in zero-shot
  methods.Our empirical evaluations show that FVDM outperforms state-of-the-art
  methods in video generation quality, while also excelling in extended tasks. By
  addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm
  in video synthesis, offering a robust framework with significant implications
  for generative modeling and multimedia applications.
paper_ai_keywords: video diffusion models (VDMs) frame-aware video diffusion model (FVDM) vectorized timestep variable (VTV) noise schedule fine-grained temporal dependencies video generation image-to-video generation video interpolation long video synthesis catastrophic forgetting zero-shot methods generative modeling multimedia applications