title: Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using
    Real-Time Warped Noise
summary: Generative modeling aims to transform random noise into structured outputs.
  In this work, we enhance video diffusion models by allowing motion control via
  structured latent noise sampling. This is achieved by just a change in data: we
  pre-process training videos to yield structured noise. Consequently, our method
  is agnostic to diffusion model design, requiring no changes to model
  architectures or training pipelines. Specifically, we propose a novel noise
  warping algorithm, fast enough to run in real time, that replaces random
  temporal Gaussianity with correlated warped noise derived from optical flow
  fields, while preserving the spatial Gaussianity. The efficiency of our
  algorithm enables us to fine-tune modern video diffusion base models using
  warped noise with minimal overhead, and provide a one-stop solution for a wide
  range of user-friendly motion control: local object motion control, global
  camera movement control, and motion transfer. The harmonization between
  temporal coherence and spatial Gaussianity in our warped noise leads to
  effective motion control while maintaining per-frame pixel quality. Extensive
  experiments and user studies demonstrate the advantages of our method, making
  it a robust and scalable approach for controlling motion in video diffusion
  models. Video results are available on our webpage:
  https://vgenai-netflix-eyeline-research.github.io/Go-with-the-Flow. Source code
  and model checkpoints are available on GitHub:
  https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.
paper_authors._id: 679086696d5aed184a333663
paper_authors.name: Ryan Burgert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:54:28.795Z
paper_authors._id: 62f69fe504e5e02f82aef7fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f69fe504e5e02f82aef7fb/m4CA0XlTKDGIg_WQVmGdv.png
paper_authors.fullname: Ryan Burgert
paper_authors.type: user
paper_authors.user: OneOverZero
paper_authors._id: 679086696d5aed184a333664
paper_authors.name: Yuancheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:54:51.597Z
paper_authors._id: 6505e2cdad3134ed7e63c8d0
paper_authors.avatarUrl: /avatars/f0e5dcf5a3deb96d2c9cb50b4433cc13.svg
paper_authors.fullname: Yuancheng Xu
paper_authors.type: user
paper_authors.user: YuanchengXu
paper_authors._id: 679086696d5aed184a333665
paper_authors.name: Wenqi Xian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:54:58.956Z
paper_authors._id: 634ada2429bfb4082451eb7d
paper_authors.avatarUrl: /avatars/be697236b58b3451c7d1aeeaf3f3c43a.svg
paper_authors.fullname: Wenqi Xian
paper_authors.type: user
paper_authors.user: wendy5823
paper_authors._id: 679086696d5aed184a333666
paper_authors.name: Oliver Pilarski
paper_authors._id: 679086696d5aed184a333667
paper_authors.name: Pascal Clausen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:55:20.961Z
paper_authors._id: 66f6dfae7024ed3df984b9da
paper_authors.avatarUrl: /avatars/7ce396a8bcee0a26d5634ea3ca2a1dbb.svg
paper_authors.fullname: Pascal Clausen
paper_authors.type: user
paper_authors.user: pascalclausen
paper_authors._id: 679086696d5aed184a333668
paper_authors.name: Mingming He
paper_authors._id: 679086696d5aed184a333669
paper_authors.name: Li Ma
paper_authors._id: 679086696d5aed184a33366a
paper_authors.name: Yitong Deng
paper_authors._id: 679086696d5aed184a33366b
paper_authors.name: Lingxiao Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-22T05:47:27.544Z
paper_authors._id: 66cb7dfcc5c61e4baa96469d
paper_authors.avatarUrl: /avatars/6cab6e06164371778d15e4a1a3278eac.svg
paper_authors.fullname: Lingxiao Li
paper_authors.type: user
paper_authors.user: lingxiaol
paper_authors._id: 679086696d5aed184a33366c
paper_authors.name: Mohsen Mousavi
paper_authors._id: 679086696d5aed184a33366d
paper_authors.name: Michael Ryoo
paper_authors._id: 679086696d5aed184a33366e
paper_authors.name: Paul Debevec
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:56:57.475Z
paper_authors._id: 646555c7fafbb0cdb40968d9
paper_authors.avatarUrl: /avatars/b6a915da510c5c405ad33354deb9b5ee.svg
paper_authors.fullname: Paul Debevec
paper_authors.type: user
paper_authors.user: debevec
paper_authors._id: 679086696d5aed184a33366f
paper_authors.name: Ning Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:18:07.559Z
paper_authors._id: 6362bcbe8f43a912fc722969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
paper_authors.fullname: Ning Yu
paper_authors.type: user
paper_authors.user: ningyu1991
paper_title: Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using
    Real-Time Warped Noise
paper_summary: Generative modeling aims to transform random noise into structured outputs.
  In this work, we enhance video diffusion models by allowing motion control via
  structured latent noise sampling. This is achieved by just a change in data: we
  pre-process training videos to yield structured noise. Consequently, our method
  is agnostic to diffusion model design, requiring no changes to model
  architectures or training pipelines. Specifically, we propose a novel noise
  warping algorithm, fast enough to run in real time, that replaces random
  temporal Gaussianity with correlated warped noise derived from optical flow
  fields, while preserving the spatial Gaussianity. The efficiency of our
  algorithm enables us to fine-tune modern video diffusion base models using
  warped noise with minimal overhead, and provide a one-stop solution for a wide
  range of user-friendly motion control: local object motion control, global
  camera movement control, and motion transfer. The harmonization between
  temporal coherence and spatial Gaussianity in our warped noise leads to
  effective motion control while maintaining per-frame pixel quality. Extensive
  experiments and user studies demonstrate the advantages of our method, making
  it a robust and scalable approach for controlling motion in video diffusion
  models. Video results are available on our webpage:
  https://vgenai-netflix-eyeline-research.github.io/Go-with-the-Flow. Source code
  and model checkpoints are available on GitHub:
  https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.
paper_ai_keywords: video diffusion models structured latent noise sampling pre-process training videos structured noise noise warping algorithm random temporal Gaussianity correlated warped noise optical flow fields spatial Gaussianity fine-tune modern video diffusion base models warped noise local object motion control global camera movement control motion transfer temporal coherence per-frame pixel quality