Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using
  Real-Time Warped Noise
Generative modeling aims to transform random noise into structured outputs.
In this work, we enhance video diffusion models by allowing motion control via
structured latent noise sampling. This is achieved by just a change in data: we
pre-process training videos to yield structured noise. Consequently, our method
is agnostic to diffusion model design, requiring no changes to model
architectures or training pipelines. Specifically, we propose a novel noise
warping algorithm, fast enough to run in real time, that replaces random
temporal Gaussianity with correlated warped noise derived from optical flow
fields, while preserving the spatial Gaussianity. The efficiency of our
algorithm enables us to fine-tune modern video diffusion base models using
warped noise with minimal overhead, and provide a one-stop solution for a wide
range of user-friendly motion control: local object motion control, global
camera movement control, and motion transfer. The harmonization between
temporal coherence and spatial Gaussianity in our warped noise leads to
effective motion control while maintaining per-frame pixel quality. Extensive
experiments and user studies demonstrate the advantages of our method, making
it a robust and scalable approach for controlling motion in video diffusion
models. Video results are available on our webpage:
https://vgenai-netflix-eyeline-research.github.io/Go-with-the-Flow. Source code
and model checkpoints are available on GitHub:
https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.
679086696d5aed184a333663
Ryan Burgert
admin_assigned
2025-01-22T15:54:28.795Z
62f69fe504e5e02f82aef7fb
https://cdn-avatars.huggingface.co/v1/production/uploads/62f69fe504e5e02f82aef7fb/m4CA0XlTKDGIg_WQVmGdv.png
Ryan Burgert
user
OneOverZero
679086696d5aed184a333664
Yuancheng Xu
admin_assigned
2025-01-22T15:54:51.597Z
6505e2cdad3134ed7e63c8d0
/avatars/f0e5dcf5a3deb96d2c9cb50b4433cc13.svg
Yuancheng Xu
user
YuanchengXu
679086696d5aed184a333665
Wenqi Xian
admin_assigned
2025-01-22T15:54:58.956Z
634ada2429bfb4082451eb7d
/avatars/be697236b58b3451c7d1aeeaf3f3c43a.svg
Wenqi Xian
user
wendy5823
679086696d5aed184a333666
Oliver Pilarski
679086696d5aed184a333667
Pascal Clausen
admin_assigned
2025-01-22T15:55:20.961Z
66f6dfae7024ed3df984b9da
/avatars/7ce396a8bcee0a26d5634ea3ca2a1dbb.svg
Pascal Clausen
user
pascalclausen
679086696d5aed184a333668
Mingming He
679086696d5aed184a333669
Li Ma
679086696d5aed184a33366a
Yitong Deng
679086696d5aed184a33366b
Lingxiao Li
extracted_pending
2025-01-22T05:47:27.544Z
66cb7dfcc5c61e4baa96469d
/avatars/6cab6e06164371778d15e4a1a3278eac.svg
Lingxiao Li
user
lingxiaol
679086696d5aed184a33366c
Mohsen Mousavi
679086696d5aed184a33366d
Michael Ryoo
679086696d5aed184a33366e
Paul Debevec
admin_assigned
2025-01-22T15:56:57.475Z
646555c7fafbb0cdb40968d9
/avatars/b6a915da510c5c405ad33354deb9b5ee.svg
Paul Debevec
user
debevec
679086696d5aed184a33366f
Ning Yu
claimed_verified
2025-01-23T09:18:07.559Z
6362bcbe8f43a912fc722969
https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
Ning Yu
user
ningyu1991
Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using
  Real-Time Warped Noise
Generative modeling aims to transform random noise into structured outputs.
In this work, we enhance video diffusion models by allowing motion control via
structured latent noise sampling. This is achieved by just a change in data: we
pre-process training videos to yield structured noise. Consequently, our method
is agnostic to diffusion model design, requiring no changes to model
architectures or training pipelines. Specifically, we propose a novel noise
warping algorithm, fast enough to run in real time, that replaces random
temporal Gaussianity with correlated warped noise derived from optical flow
fields, while preserving the spatial Gaussianity. The efficiency of our
algorithm enables us to fine-tune modern video diffusion base models using
warped noise with minimal overhead, and provide a one-stop solution for a wide
range of user-friendly motion control: local object motion control, global
camera movement control, and motion transfer. The harmonization between
temporal coherence and spatial Gaussianity in our warped noise leads to
effective motion control while maintaining per-frame pixel quality. Extensive
experiments and user studies demonstrate the advantages of our method, making
it a robust and scalable approach for controlling motion in video diffusion
models. Video results are available on our webpage:
https://vgenai-netflix-eyeline-research.github.io/Go-with-the-Flow. Source code
and model checkpoints are available on GitHub:
https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.
A new noise warping algorithm improves motion control in video diffusion models by transforming random temporal noise into correlated noise using optical flow, enabling real-time fine-tuning and scalability.
video diffusion models structured latent noise noise warping algorithm optical flow fields temporal coherence spatial Gaussianity local object motion control global camera movement control motion transfer