title: TRIP: Temporal Residual Learning with Image Noise Prior for
    Image-to-Video Diffusion Models
summary: Recent advances in text-to-video generation have demonstrated the utility of
  powerful diffusion models. Nevertheless, the problem is not trivial when
  shaping diffusion models to animate static image (i.e., image-to-video
  generation). The difficulty originates from the aspect that the diffusion
  process of subsequent animated frames should not only preserve the faithful
  alignment with the given image but also pursue temporal coherence among
  adjacent frames. To alleviate this, we present TRIP, a new recipe of
  image-to-video diffusion paradigm that pivots on image noise prior derived from
  static image to jointly trigger inter-frame relational reasoning and ease the
  coherent temporal modeling via temporal residual learning. Technically, the
  image noise prior is first attained through one-step backward diffusion process
  based on both static image and noised video latent codes. Next, TRIP executes a
  residual-like dual-path scheme for noise prediction: 1) a shortcut path that
  directly takes image noise prior as the reference noise of each frame to
  amplify the alignment between the first frame and subsequent frames; 2) a
  residual path that employs 3D-UNet over noised video and static image latent
  codes to enable inter-frame relational reasoning, thereby easing the learning
  of the residual noise for each frame. Furthermore, both reference and residual
  noise of each frame are dynamically merged via attention mechanism for final
  video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
  datasets demonstrate the effectiveness of our TRIP for image-to-video
  generation. Please see our project page at https://trip-i2v.github.io/TRIP/.
paper_authors._id: 66024403a7a608d711560671
paper_authors.name: Zhongwei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T15:51:02.729Z
paper_authors._id: 6496f5754a3c31df8e3139f6
paper_authors.avatarUrl: /avatars/cf789d1986f976373c82b2976df4542a.svg
paper_authors.fullname: Zhongwei Zhang
paper_authors.type: user
paper_authors.user: zzwustc
paper_authors._id: 66024403a7a608d711560672
paper_authors.name: Fuchen Long
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T09:41:37.428Z
paper_authors._id: 6449f2dfeb7db8f70fb990f8
paper_authors.avatarUrl: /avatars/72c02e754bb35dab05c4a5f1e69c95f1.svg
paper_authors.fullname: Fuchen
paper_authors.type: user
paper_authors.user: FireCRT
paper_authors._id: 66024403a7a608d711560673
paper_authors.name: Yingwei Pan
paper_authors._id: 66024403a7a608d711560674
paper_authors.name: Zhaofan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:33:04.386Z
paper_authors._id: 64ac099294ae6b609d0f0713
paper_authors.avatarUrl: /avatars/7f50e3e1597fb5fd0dad9fe7fd2e26ec.svg
paper_authors.fullname: Zhaofan Qiu
paper_authors.type: user
paper_authors.user: qiudavy
paper_authors._id: 66024403a7a608d711560675
paper_authors.name: Ting Yao
paper_authors._id: 66024403a7a608d711560676
paper_authors.name: Yang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:33:48.933Z
paper_authors._id: 647d893bcfca67bc50f8974f
paper_authors.avatarUrl: /avatars/fab275f968b7d648dbdf8485c9279fb6.svg
paper_authors.fullname: Yang Cao
paper_authors.type: user
paper_authors.user: YangCaoCS
paper_authors._id: 66024403a7a608d711560677
paper_authors.name: Tao Mei
paper_title: TRIP: Temporal Residual Learning with Image Noise Prior for
    Image-to-Video Diffusion Models
paper_summary: Recent advances in text-to-video generation have demonstrated the utility of
  powerful diffusion models. Nevertheless, the problem is not trivial when
  shaping diffusion models to animate static image (i.e., image-to-video
  generation). The difficulty originates from the aspect that the diffusion
  process of subsequent animated frames should not only preserve the faithful
  alignment with the given image but also pursue temporal coherence among
  adjacent frames. To alleviate this, we present TRIP, a new recipe of
  image-to-video diffusion paradigm that pivots on image noise prior derived from
  static image to jointly trigger inter-frame relational reasoning and ease the
  coherent temporal modeling via temporal residual learning. Technically, the
  image noise prior is first attained through one-step backward diffusion process
  based on both static image and noised video latent codes. Next, TRIP executes a
  residual-like dual-path scheme for noise prediction: 1) a shortcut path that
  directly takes image noise prior as the reference noise of each frame to
  amplify the alignment between the first frame and subsequent frames; 2) a
  residual path that employs 3D-UNet over noised video and static image latent
  codes to enable inter-frame relational reasoning, thereby easing the learning
  of the residual noise for each frame. Furthermore, both reference and residual
  noise of each frame are dynamically merged via attention mechanism for final
  video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
  datasets demonstrate the effectiveness of our TRIP for image-to-video
  generation. Please see our project page at https://trip-i2v.github.io/TRIP/.
paper_ai_summary: TRIP enhances image-to-video generation using diffusion models with an image noise prior and temporal residual learning to maintain coherence and alignment.
paper_ai_keywords: diffusion models image-to-video generation image noise prior temporal coherence inter-frame relational reasoning residual learning 3D-UNet latent codes attention mechanism WebVid-10M DTDB MSR-VTT