title: Cavia: Camera-controllable Multi-view Video Diffusion with
    View-Integrated Attention
summary: In recent years there have been remarkable breakthroughs in image-to-video
  generation. However, the 3D consistency and camera controllability of generated
  frames have remained unsolved. Recent studies have attempted to incorporate
  camera control into the generation process, but their results are often limited
  to simple trajectories or lack the ability to generate consistent videos from
  multiple distinct camera paths for the same scene. To address these
  limitations, we introduce Cavia, a novel framework for camera-controllable,
  multi-view video generation, capable of converting an input image into multiple
  spatiotemporally consistent videos. Our framework extends the spatial and
  temporal attention modules into view-integrated attention modules, improving
  both viewpoint and temporal consistency. This flexible design allows for joint
  training with diverse curated data sources, including scene-level static
  videos, object-level synthetic multi-view dynamic videos, and real-world
  monocular dynamic videos. To our best knowledge, Cavia is the first of its kind
  that allows the user to precisely specify camera motion while obtaining object
  motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art
  methods in terms of geometric consistency and perceptual quality. Project Page:
  https://ir1d.github.io/Cavia/
paper_authors._id: 670de040ae6e15b485712a81
paper_authors.name: Dejia Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:10:40.250Z
paper_authors._id: 62f687a0c58915315c4ff75d
paper_authors.avatarUrl: /avatars/b657180c7666735062782edd4f6a69c9.svg
paper_authors.fullname: Dejia Xu
paper_authors.type: user
paper_authors.user: ir1d
paper_authors._id: 670de040ae6e15b485712a82
paper_authors.name: Yifan Jiang
paper_authors._id: 670de040ae6e15b485712a83
paper_authors.name: Chen Huang
paper_authors._id: 670de040ae6e15b485712a84
paper_authors.name: Liangchen Song
paper_authors._id: 670de040ae6e15b485712a85
paper_authors.name: Thorsten Gernoth
paper_authors._id: 670de040ae6e15b485712a86
paper_authors.name: Liangliang Cao
paper_authors._id: 670de040ae6e15b485712a87
paper_authors.name: Zhangyang Wang
paper_authors._id: 670de040ae6e15b485712a88
paper_authors.name: Hao Tang
paper_title: Cavia: Camera-controllable Multi-view Video Diffusion with
    View-Integrated Attention
paper_summary: In recent years there have been remarkable breakthroughs in image-to-video
  generation. However, the 3D consistency and camera controllability of generated
  frames have remained unsolved. Recent studies have attempted to incorporate
  camera control into the generation process, but their results are often limited
  to simple trajectories or lack the ability to generate consistent videos from
  multiple distinct camera paths for the same scene. To address these
  limitations, we introduce Cavia, a novel framework for camera-controllable,
  multi-view video generation, capable of converting an input image into multiple
  spatiotemporally consistent videos. Our framework extends the spatial and
  temporal attention modules into view-integrated attention modules, improving
  both viewpoint and temporal consistency. This flexible design allows for joint
  training with diverse curated data sources, including scene-level static
  videos, object-level synthetic multi-view dynamic videos, and real-world
  monocular dynamic videos. To our best knowledge, Cavia is the first of its kind
  that allows the user to precisely specify camera motion while obtaining object
  motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art
  methods in terms of geometric consistency and perceptual quality. Project Page:
  https://ir1d.github.io/Cavia/
paper_ai_summary: Cavia is a novel framework that integrates view-integrated attention modules to generate camera-controllable, multi-view, spatiotemporally consistent videos from images.
paper_ai_keywords: camera-controllable multi-view video generation view-integrated attention modules viewpoint and temporal consistency scene-level static videos object-level synthetic multi-view dynamic videos real-world monocular dynamic videos geometric consistency perceptual quality