title: VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
summary: Modern text-to-video synthesis models demonstrate coherent, photorealistic
  generation of complex videos from a text description. However, most existing
  models lack fine-grained control over camera movement, which is critical for
  downstream applications related to content creation, visual effects, and 3D
  vision. Recently, new methods demonstrate the ability to generate videos with
  controllable camera poses these techniques leverage pre-trained U-Net-based
  diffusion models that explicitly disentangle spatial and temporal generation.
  Still, no existing approach enables camera control for new, transformer-based
  video diffusion models that process spatial and temporal information jointly.
  Here, we propose to tame video transformers for 3D camera control using a
  ControlNet-like conditioning mechanism that incorporates spatiotemporal camera
  embeddings based on Plucker coordinates. The approach demonstrates
  state-of-the-art performance for controllable video generation after
  fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our
  work is the first to enable camera control for transformer-based video
  diffusion models.
paper_authors._id: 66987680f812d48679cfa868
paper_authors.name: Sherwin Bahmani
paper_authors._id: 66987680f812d48679cfa869
paper_authors.name: Ivan Skorokhodov
paper_authors._id: 66987680f812d48679cfa86a
paper_authors.name: Aliaksandr Siarohin
paper_authors._id: 66987680f812d48679cfa86b
paper_authors.name: Willi Menapace
paper_authors._id: 66987680f812d48679cfa86c
paper_authors.name: Guocheng Qian
paper_authors._id: 66987680f812d48679cfa86d
paper_authors.name: Michael Vasilkovsky
paper_authors._id: 66987680f812d48679cfa86e
paper_authors.name: Hsin-Ying Lee
paper_authors._id: 66987680f812d48679cfa86f
paper_authors.name: Chaoyang Wang
paper_authors._id: 66987680f812d48679cfa870
paper_authors.name: Jiaxu Zou
paper_authors._id: 66987680f812d48679cfa871
paper_authors.name: Andrea Tagliasacchi
paper_authors._id: 66987680f812d48679cfa872
paper_authors.name: David B. Lindell
paper_authors._id: 66987680f812d48679cfa873
paper_authors.name: Sergey Tulyakov
paper_title: VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
paper_summary: Modern text-to-video synthesis models demonstrate coherent, photorealistic
  generation of complex videos from a text description. However, most existing
  models lack fine-grained control over camera movement, which is critical for
  downstream applications related to content creation, visual effects, and 3D
  vision. Recently, new methods demonstrate the ability to generate videos with
  controllable camera poses these techniques leverage pre-trained U-Net-based
  diffusion models that explicitly disentangle spatial and temporal generation.
  Still, no existing approach enables camera control for new, transformer-based
  video diffusion models that process spatial and temporal information jointly.
  Here, we propose to tame video transformers for 3D camera control using a
  ControlNet-like conditioning mechanism that incorporates spatiotemporal camera
  embeddings based on Plucker coordinates. The approach demonstrates
  state-of-the-art performance for controllable video generation after
  fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our
  work is the first to enable camera control for transformer-based video
  diffusion models.
paper_ai_keywords: U-Net-based diffusion models spatiotemporal generation video transformers ControlNet-like conditioning mechanism Plucker coordinates controllable video generation fine-tuning RealEstate10K dataset