TRIP: Temporal Residual Learning with Image Noise Prior for
  Image-to-Video Diffusion Models
Recent advances in text-to-video generation have demonstrated the utility of
powerful diffusion models. Nevertheless, the problem is not trivial when
shaping diffusion models to animate static image (i.e., image-to-video
generation). The difficulty originates from the aspect that the diffusion
process of subsequent animated frames should not only preserve the faithful
alignment with the given image but also pursue temporal coherence among
adjacent frames. To alleviate this, we present TRIP, a new recipe of
image-to-video diffusion paradigm that pivots on image noise prior derived from
static image to jointly trigger inter-frame relational reasoning and ease the
coherent temporal modeling via temporal residual learning. Technically, the
image noise prior is first attained through one-step backward diffusion process
based on both static image and noised video latent codes. Next, TRIP executes a
residual-like dual-path scheme for noise prediction: 1) a shortcut path that
directly takes image noise prior as the reference noise of each frame to
amplify the alignment between the first frame and subsequent frames; 2) a
residual path that employs 3D-UNet over noised video and static image latent
codes to enable inter-frame relational reasoning, thereby easing the learning
of the residual noise for each frame. Furthermore, both reference and residual
noise of each frame are dynamically merged via attention mechanism for final
video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
datasets demonstrate the effectiveness of our TRIP for image-to-video
generation. Please see our project page at https://trip-i2v.github.io/TRIP/.
66024403a7a608d711560671
Zhongwei Zhang
claimed_verified
2025-05-27T15:51:02.729Z
6496f5754a3c31df8e3139f6
/avatars/cf789d1986f976373c82b2976df4542a.svg
Zhongwei Zhang
user
zzwustc
66024403a7a608d711560672
Fuchen Long
claimed_verified
2025-01-16T09:41:37.428Z
6449f2dfeb7db8f70fb990f8
/avatars/72c02e754bb35dab05c4a5f1e69c95f1.svg
Fuchen
user
FireCRT
66024403a7a608d711560673
Yingwei Pan
66024403a7a608d711560674
Zhaofan Qiu
admin_assigned
2024-03-26T09:33:04.386Z
64ac099294ae6b609d0f0713
/avatars/7f50e3e1597fb5fd0dad9fe7fd2e26ec.svg
Zhaofan Qiu
user
qiudavy
66024403a7a608d711560675
Ting Yao
66024403a7a608d711560676
Yang Cao
admin_assigned
2024-03-26T09:33:48.933Z
647d893bcfca67bc50f8974f
/avatars/fab275f968b7d648dbdf8485c9279fb6.svg
Yang Cao
user
YangCaoCS
66024403a7a608d711560677
Tao Mei
TRIP: Temporal Residual Learning with Image Noise Prior for
  Image-to-Video Diffusion Models
Recent advances in text-to-video generation have demonstrated the utility of
powerful diffusion models. Nevertheless, the problem is not trivial when
shaping diffusion models to animate static image (i.e., image-to-video
generation). The difficulty originates from the aspect that the diffusion
process of subsequent animated frames should not only preserve the faithful
alignment with the given image but also pursue temporal coherence among
adjacent frames. To alleviate this, we present TRIP, a new recipe of
image-to-video diffusion paradigm that pivots on image noise prior derived from
static image to jointly trigger inter-frame relational reasoning and ease the
coherent temporal modeling via temporal residual learning. Technically, the
image noise prior is first attained through one-step backward diffusion process
based on both static image and noised video latent codes. Next, TRIP executes a
residual-like dual-path scheme for noise prediction: 1) a shortcut path that
directly takes image noise prior as the reference noise of each frame to
amplify the alignment between the first frame and subsequent frames; 2) a
residual path that employs 3D-UNet over noised video and static image latent
codes to enable inter-frame relational reasoning, thereby easing the learning
of the residual noise for each frame. Furthermore, both reference and residual
noise of each frame are dynamically merged via attention mechanism for final
video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
datasets demonstrate the effectiveness of our TRIP for image-to-video
generation. Please see our project page at https://trip-i2v.github.io/TRIP/.
TRIP enhances image-to-video generation using diffusion models with an image noise prior and temporal residual learning to maintain coherence and alignment.
diffusion models image-to-video generation image noise prior temporal coherence inter-frame relational reasoning residual learning 3D-UNet latent codes attention mechanism WebVid-10M DTDB MSR-VTT