History-Guided Video Diffusion
Classifier-free guidance (CFG) is a key technique for improving conditional
generation in diffusion models, enabling more accurate control while enhancing
sample quality. It is natural to extend this technique to video diffusion,
which generates video conditioned on a variable number of context frames,
collectively referred to as history. However, we find two key challenges to
guiding with variable-length history: architectures that only support
fixed-size conditioning, and the empirical observation that CFG-style history
dropout performs poorly. To address this, we propose the Diffusion Forcing
Transformer (DFoT), a video diffusion architecture and theoretically grounded
training objective that jointly enable conditioning on a flexible number of
history frames. We then introduce History Guidance, a family of guidance
methods uniquely enabled by DFoT. We show that its simplest form, vanilla
history guidance, already significantly improves video generation quality and
temporal consistency. A more advanced method, history guidance across time and
frequency further enhances motion dynamics, enables compositional
generalization to out-of-distribution history, and can stably roll out
extremely long videos. Website: https://boyuan.space/history-guidance
67aac6052c02e43558b6b4b0
Kiwhan Song
claimed_verified
2025-02-12T09:16:57.802Z
6613663bcfbba5e761a69531
/avatars/2baf348371d87a2f9dd4b9c56f1483a9.svg
Kiwhan Song
user
kiwhansong
67aac6052c02e43558b6b4b1
Boyuan Chen
claimed_verified
2025-02-18T09:34:45.325Z
646935e05d70156639500beb
/avatars/1373ef2a40145aa505a03bfbae37c95a.svg
Boyuan Chen
user
buoyancy99
67aac6052c02e43558b6b4b2
Max Simchowitz
67aac6052c02e43558b6b4b3
Yilun Du
67aac6052c02e43558b6b4b4
Russ Tedrake
67aac6052c02e43558b6b4b5
Vincent Sitzmann
History-Guided Video Diffusion
Classifier-free guidance (CFG) is a key technique for improving conditional
generation in diffusion models, enabling more accurate control while enhancing
sample quality. It is natural to extend this technique to video diffusion,
which generates video conditioned on a variable number of context frames,
collectively referred to as history. However, we find two key challenges to
guiding with variable-length history: architectures that only support
fixed-size conditioning, and the empirical observation that CFG-style history
dropout performs poorly. To address this, we propose the Diffusion Forcing
Transformer (DFoT), a video diffusion architecture and theoretically grounded
training objective that jointly enable conditioning on a flexible number of
history frames. We then introduce History Guidance, a family of guidance
methods uniquely enabled by DFoT. We show that its simplest form, vanilla
history guidance, already significantly improves video generation quality and
temporal consistency. A more advanced method, history guidance across time and
frequency further enhances motion dynamics, enables compositional
generalization to out-of-distribution history, and can stably roll out
extremely long videos. Website: https://boyuan.space/history-guidance
A novel video diffusion architecture, DFoT, and associated guidance methods improve video generation quality and temporal consistency by enabling flexible conditioning on variable-length history frames.
classifier-free guidance diffusion models video diffusion context frames history fixed-size conditioning history dropout Diffusion Forcing Transformer DFoT History Guidance temporal consistency motion dynamics compositional generalization long videos