title: CamViG: Camera Aware Image-to-Video Generation with Multimodal
    Transformers
summary: We extend multimodal transformers to include 3D camera motion as a
  conditioning signal for the task of video generation. Generative video models
  are becoming increasingly powerful, thus focusing research efforts on methods
  of controlling the output of such models. We propose to add virtual 3D camera
  controls to generative video methods by conditioning generated video on an
  encoding of three-dimensional camera movement over the course of the generated
  video. Results demonstrate that we are (1) able to successfully control the
  camera during video generation, starting from a single frame and a camera
  signal, and (2) we demonstrate the accuracy of the generated 3D camera paths
  using traditional computer vision methods.
paper_authors._id: 66500d3ad5bea69bca0726eb
paper_authors.name: Andrew Marmon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:05:40.878Z
paper_authors._id: 60ac2ec408805574fcf076f4
paper_authors.avatarUrl: /avatars/a8cd60f39ba89939c5841252dc9d742b.svg
paper_authors.fullname: Andrew Marmon
paper_authors.type: user
paper_authors.user: acoadmarmon
paper_authors._id: 66500d3ad5bea69bca0726ec
paper_authors.name: Grant Schindler
paper_authors._id: 66500d3ad5bea69bca0726ed
paper_authors.name: José Lezama
paper_authors._id: 66500d3ad5bea69bca0726ee
paper_authors.name: Dan Kondratyuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:05:55.421Z
paper_authors._id: 63351623259c518276f0c63f
paper_authors.avatarUrl: /avatars/77a49a1addeec11c9a799f9590f181b1.svg
paper_authors.fullname: Dan Kondratyuk
paper_authors.type: user
paper_authors.user: hyperparticle
paper_authors._id: 66500d3ad5bea69bca0726ef
paper_authors.name: Bryan Seybold
paper_authors._id: 66500d3ad5bea69bca0726f0
paper_authors.name: Irfan Essa
paper_title: CamViG: Camera Aware Image-to-Video Generation with Multimodal
    Transformers
paper_summary: We extend multimodal transformers to include 3D camera motion as a
  conditioning signal for the task of video generation. Generative video models
  are becoming increasingly powerful, thus focusing research efforts on methods
  of controlling the output of such models. We propose to add virtual 3D camera
  controls to generative video methods by conditioning generated video on an
  encoding of three-dimensional camera movement over the course of the generated
  video. Results demonstrate that we are (1) able to successfully control the
  camera during video generation, starting from a single frame and a camera
  signal, and (2) we demonstrate the accuracy of the generated 3D camera paths
  using traditional computer vision methods.
paper_ai_keywords: multimodal transformers 3D camera motion conditioning signal video generation generative video models virtual 3D camera controls encoding of three-dimensional camera movement generated 3D camera paths