VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Modern text-to-video synthesis models demonstrate coherent, photorealistic
generation of complex videos from a text description. However, most existing
models lack fine-grained control over camera movement, which is critical for
downstream applications related to content creation, visual effects, and 3D
vision. Recently, new methods demonstrate the ability to generate videos with
controllable camera poses these techniques leverage pre-trained U-Net-based
diffusion models that explicitly disentangle spatial and temporal generation.
Still, no existing approach enables camera control for new, transformer-based
video diffusion models that process spatial and temporal information jointly.
Here, we propose to tame video transformers for 3D camera control using a
ControlNet-like conditioning mechanism that incorporates spatiotemporal camera
embeddings based on Plucker coordinates. The approach demonstrates
state-of-the-art performance for controllable video generation after
fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our
work is the first to enable camera control for transformer-based video
diffusion models.
66987680f812d48679cfa868
Sherwin Bahmani
66987680f812d48679cfa869
Ivan Skorokhodov
66987680f812d48679cfa86a
Aliaksandr Siarohin
66987680f812d48679cfa86b
Willi Menapace
66987680f812d48679cfa86c
Guocheng Qian
66987680f812d48679cfa86d
Michael Vasilkovsky
66987680f812d48679cfa86e
Hsin-Ying Lee
66987680f812d48679cfa86f
Chaoyang Wang
66987680f812d48679cfa870
Jiaxu Zou
66987680f812d48679cfa871
Andrea Tagliasacchi
66987680f812d48679cfa872
David B. Lindell
66987680f812d48679cfa873
Sergey Tulyakov
VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Modern text-to-video synthesis models demonstrate coherent, photorealistic
generation of complex videos from a text description. However, most existing
models lack fine-grained control over camera movement, which is critical for
downstream applications related to content creation, visual effects, and 3D
vision. Recently, new methods demonstrate the ability to generate videos with
controllable camera poses these techniques leverage pre-trained U-Net-based
diffusion models that explicitly disentangle spatial and temporal generation.
Still, no existing approach enables camera control for new, transformer-based
video diffusion models that process spatial and temporal information jointly.
Here, we propose to tame video transformers for 3D camera control using a
ControlNet-like conditioning mechanism that incorporates spatiotemporal camera
embeddings based on Plucker coordinates. The approach demonstrates
state-of-the-art performance for controllable video generation after
fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our
work is the first to enable camera control for transformer-based video
diffusion models.
A method for controlling camera movement in transformer-based video diffusion models using spatiotemporal embeddings achieves state-of-the-art performance.
U-Net-based diffusion models disentanglement spatial generation temporal generation ControlNet-like conditioning mechanism Plucker coordinates fine-tuning RealEstate10K dataset transformer-based video diffusion models