CamViG: Camera Aware Image-to-Video Generation with Multimodal
  Transformers
We extend multimodal transformers to include 3D camera motion as a
conditioning signal for the task of video generation. Generative video models
are becoming increasingly powerful, thus focusing research efforts on methods
of controlling the output of such models. We propose to add virtual 3D camera
controls to generative video methods by conditioning generated video on an
encoding of three-dimensional camera movement over the course of the generated
video. Results demonstrate that we are (1) able to successfully control the
camera during video generation, starting from a single frame and a camera
signal, and (2) we demonstrate the accuracy of the generated 3D camera paths
using traditional computer vision methods.
66500d3ad5bea69bca0726eb
Andrew Marmon
admin_assigned
2024-05-24T12:05:40.878Z
60ac2ec408805574fcf076f4
/avatars/a8cd60f39ba89939c5841252dc9d742b.svg
Andrew Marmon
user
acoadmarmon
66500d3ad5bea69bca0726ec
Grant Schindler
66500d3ad5bea69bca0726ed
José Lezama
66500d3ad5bea69bca0726ee
Dan Kondratyuk
admin_assigned
2024-05-24T12:05:55.421Z
63351623259c518276f0c63f
/avatars/77a49a1addeec11c9a799f9590f181b1.svg
Dan Kondratyuk
user
hyperparticle
66500d3ad5bea69bca0726ef
Bryan Seybold
66500d3ad5bea69bca0726f0
Irfan Essa
CamViG: Camera Aware Image-to-Video Generation with Multimodal
  Transformers
We extend multimodal transformers to include 3D camera motion as a
conditioning signal for the task of video generation. Generative video models
are becoming increasingly powerful, thus focusing research efforts on methods
of controlling the output of such models. We propose to add virtual 3D camera
controls to generative video methods by conditioning generated video on an
encoding of three-dimensional camera movement over the course of the generated
video. Results demonstrate that we are (1) able to successfully control the
camera during video generation, starting from a single frame and a camera
signal, and (2) we demonstrate the accuracy of the generated 3D camera paths
using traditional computer vision methods.
Generative video models are enhanced with 3D camera motion conditioning to control and accurately simulate camera movements during video generation.
multimodal transformers 3D camera motion conditioning signal generative video models camera signal 3D camera paths