Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient
  Video Latent Generation
Video generation using diffusion-based models is constrained by high
computational costs due to the frame-wise iterative diffusion process. This
work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent
video generation. Our key discovery is that coarse-grained noises in earlier
denoising steps have demonstrated high motion consistency across consecutive
video frames. Following this observation, Dr. Mo propagates those
coarse-grained noises onto the next frame by incorporating carefully designed,
lightweight inter-frame motions, eliminating massive computational redundancy
in frame-wise diffusion models. The more sensitive and fine-grained noises are
still acquired via later denoising steps, which can be essential to retain
visual qualities. As such, deciding which intermediate steps should switch from
motion-based propagations to denoising can be a crucial problem and a key
tradeoff between efficiency and quality. Dr. Mo employs a meta-network named
Denoising Step Selector (DSS) to dynamically determine desirable intermediate
steps across video frames. Extensive evaluations on video generation and
editing tasks have shown that Dr. Mo can substantially accelerate diffusion
models in video tasks with improved visual qualities.
66ecd648dca68f236c888ecc
Chenyu Wang
66ecd648dca68f236c888ecd
Shuo Yan
66ecd648dca68f236c888ece
Yixuan Chen
66ecd648dca68f236c888ecf
Yujiang Wang
66ecd648dca68f236c888ed0
Mingzhi Dong
66ecd648dca68f236c888ed1
Xiaochen Yang
admin_assigned
2024-09-20T09:41:31.943Z
6654140eff66ec2867458ced
/avatars/6c34055ea1aebe015ac4a9b85969e94d.svg
Xiaochen Yang
user
genye
66ecd648dca68f236c888ed2
Dongsheng Li
admin_assigned
2024-09-20T09:42:14.989Z
6524f13821af4b07309015aa
/avatars/f97d6a041428d48305568df7afd83cc8.svg
dongsheng li
user
dongshengli
66ecd648dca68f236c888ed3
Robert P. Dick
66ecd648dca68f236c888ed4
Qin Lv
66ecd648dca68f236c888ed5
Fan Yang
66ecd648dca68f236c888ed6
Tun Lu
66ecd648dca68f236c888ed7
Ning Gu
66ecd648dca68f236c888ed8
Li Shang
Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient
  Video Latent Generation
Video generation using diffusion-based models is constrained by high
computational costs due to the frame-wise iterative diffusion process. This
work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent
video generation. Our key discovery is that coarse-grained noises in earlier
denoising steps have demonstrated high motion consistency across consecutive
video frames. Following this observation, Dr. Mo propagates those
coarse-grained noises onto the next frame by incorporating carefully designed,
lightweight inter-frame motions, eliminating massive computational redundancy
in frame-wise diffusion models. The more sensitive and fine-grained noises are
still acquired via later denoising steps, which can be essential to retain
visual qualities. As such, deciding which intermediate steps should switch from
motion-based propagations to denoising can be a crucial problem and a key
tradeoff between efficiency and quality. Dr. Mo employs a meta-network named
Denoising Step Selector (DSS) to dynamically determine desirable intermediate
steps across video frames. Extensive evaluations on video generation and
editing tasks have shown that Dr. Mo can substantially accelerate diffusion
models in video tasks with improved visual qualities.
Dr. Mo accelerates latent video generation by reusing coarse-grained noise from earlier denoising steps with a meta-network, maintaining visual quality and efficiency.
diffusion-based models latent video generation frame-wise iterative diffusion coarse-grained noises motion consistency inter-frame motions denoising steps Denoising Step Selector (DSS)