Cavia: Camera-controllable Multi-view Video Diffusion with
  View-Integrated Attention
In recent years there have been remarkable breakthroughs in image-to-video
generation. However, the 3D consistency and camera controllability of generated
frames have remained unsolved. Recent studies have attempted to incorporate
camera control into the generation process, but their results are often limited
to simple trajectories or lack the ability to generate consistent videos from
multiple distinct camera paths for the same scene. To address these
limitations, we introduce Cavia, a novel framework for camera-controllable,
multi-view video generation, capable of converting an input image into multiple
spatiotemporally consistent videos. Our framework extends the spatial and
temporal attention modules into view-integrated attention modules, improving
both viewpoint and temporal consistency. This flexible design allows for joint
training with diverse curated data sources, including scene-level static
videos, object-level synthetic multi-view dynamic videos, and real-world
monocular dynamic videos. To our best knowledge, Cavia is the first of its kind
that allows the user to precisely specify camera motion while obtaining object
motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art
methods in terms of geometric consistency and perceptual quality. Project Page:
https://ir1d.github.io/Cavia/
670de040ae6e15b485712a81
Dejia Xu
claimed_verified
2024-10-15T09:10:40.250Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
670de040ae6e15b485712a82
Yifan Jiang
670de040ae6e15b485712a83
Chen Huang
670de040ae6e15b485712a84
Liangchen Song
670de040ae6e15b485712a85
Thorsten Gernoth
670de040ae6e15b485712a86
Liangliang Cao
670de040ae6e15b485712a87
Zhangyang Wang
670de040ae6e15b485712a88
Hao Tang
Cavia: Camera-controllable Multi-view Video Diffusion with
  View-Integrated Attention
In recent years there have been remarkable breakthroughs in image-to-video
generation. However, the 3D consistency and camera controllability of generated
frames have remained unsolved. Recent studies have attempted to incorporate
camera control into the generation process, but their results are often limited
to simple trajectories or lack the ability to generate consistent videos from
multiple distinct camera paths for the same scene. To address these
limitations, we introduce Cavia, a novel framework for camera-controllable,
multi-view video generation, capable of converting an input image into multiple
spatiotemporally consistent videos. Our framework extends the spatial and
temporal attention modules into view-integrated attention modules, improving
both viewpoint and temporal consistency. This flexible design allows for joint
training with diverse curated data sources, including scene-level static
videos, object-level synthetic multi-view dynamic videos, and real-world
monocular dynamic videos. To our best knowledge, Cavia is the first of its kind
that allows the user to precisely specify camera motion while obtaining object
motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art
methods in terms of geometric consistency and perceptual quality. Project Page:
https://ir1d.github.io/Cavia/
Cavia is a novel framework that integrates view-integrated attention modules to generate camera-controllable, multi-view, spatiotemporally consistent videos from images.
camera-controllable multi-view video generation view-integrated attention modules viewpoint and temporal consistency scene-level static videos object-level synthetic multi-view dynamic videos real-world monocular dynamic videos geometric consistency perceptual quality