title: CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
summary: Recently video diffusion models have emerged as expressive generative tools
  for high-quality video content creation readily available to general users.
  However, these models often do not offer precise control over camera poses for
  video generation, limiting the expression of cinematic language and user
  control. To address this issue, we introduce CamCo, which allows fine-grained
  Camera pose Control for image-to-video generation. We equip a pre-trained
  image-to-video generator with accurately parameterized camera pose input using
  Pl\"ucker coordinates. To enhance 3D consistency in the videos produced, we
  integrate an epipolar attention module in each attention block that enforces
  epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on
  real-world videos with camera poses estimated through structure-from-motion
  algorithms to better synthesize object motion. Our experiments show that CamCo
  significantly improves 3D consistency and camera control capabilities compared
  to previous models while effectively generating plausible object motion.
  Project page: https://ir1d.github.io/CamCo/
paper_authors._id: 665fe43a79373fff9c0f0781
paper_authors.name: Dejia Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T10:45:22.237Z
paper_authors._id: 62f687a0c58915315c4ff75d
paper_authors.avatarUrl: /avatars/b657180c7666735062782edd4f6a69c9.svg
paper_authors.fullname: Dejia Xu
paper_authors.type: user
paper_authors.user: ir1d
paper_authors._id: 665fe43a79373fff9c0f0782
paper_authors.name: Weili Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T10:45:28.607Z
paper_authors._id: 64c1a69e226e016da8450ae2
paper_authors.avatarUrl: /avatars/54c161e8b8543244ed13cbe47017624e.svg
paper_authors.fullname: Weili Nie
paper_authors.type: user
paper_authors.user: xiaoli08
paper_authors._id: 665fe43a79373fff9c0f0783
paper_authors.name: Chao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T10:45:54.513Z
paper_authors._id: 63420d0706a304bdc4365aac
paper_authors.avatarUrl: /avatars/06fe5b60a197ddf13538dde063b1eb0a.svg
paper_authors.fullname: Chao Liu
paper_authors.type: user
paper_authors.user: asroy
paper_authors._id: 665fe43a79373fff9c0f0784
paper_authors.name: Sifei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T10:46:38.560Z
paper_authors._id: 62fab69f8cd542e895bafd6e
paper_authors.avatarUrl: /avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
paper_authors.fullname: Sifei Liu
paper_authors.type: user
paper_authors.user: zwrq
paper_authors._id: 665fe43a79373fff9c0f0785
paper_authors.name: Jan Kautz
paper_authors._id: 665fe43a79373fff9c0f0786
paper_authors.name: Zhangyang Wang
paper_authors._id: 665fe43a79373fff9c0f0787
paper_authors.name: Arash Vahdat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T10:47:00.058Z
paper_authors._id: 62f6956cd3bdacb7eec02920
paper_authors.avatarUrl: /avatars/b22db0823311f866c00db2efc4b9f814.svg
paper_authors.fullname: Arash Vahdat
paper_authors.type: user
paper_authors.user: avahdat
paper_title: CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
paper_summary: Recently video diffusion models have emerged as expressive generative tools
  for high-quality video content creation readily available to general users.
  However, these models often do not offer precise control over camera poses for
  video generation, limiting the expression of cinematic language and user
  control. To address this issue, we introduce CamCo, which allows fine-grained
  Camera pose Control for image-to-video generation. We equip a pre-trained
  image-to-video generator with accurately parameterized camera pose input using
  Pl\"ucker coordinates. To enhance 3D consistency in the videos produced, we
  integrate an epipolar attention module in each attention block that enforces
  epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on
  real-world videos with camera poses estimated through structure-from-motion
  algorithms to better synthesize object motion. Our experiments show that CamCo
  significantly improves 3D consistency and camera control capabilities compared
  to previous models while effectively generating plausible object motion.
  Project page: https://ir1d.github.io/CamCo/
paper_ai_summary: CamCo enhances camera pose control in video generation by using Plücker coordinates and an epipolar attention module, improving 3D consistency and object motion synthesis.
paper_ai_keywords: video diffusion models image-to-video generator Plücker coordinates epipolar attention module epipolar constraints feature maps structure-from-motion 3D consistency camera control object motion synthesis