title: Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric
    Diffusion
summary: Current methods for 3D scene reconstruction from sparse posed images employ
  intermediate 3D representations such as neural fields, voxel grids, or 3D
  Gaussians, to achieve multi-view consistent scene appearance and geometry. In
  this paper we introduce MVGD, a diffusion-based architecture capable of direct
  pixel-level generation of images and depth maps from novel viewpoints, given an
  arbitrary number of input views. Our method uses raymap conditioning to both
  augment visual features with spatial information from different viewpoints, as
  well as to guide the generation of images and depth maps from novel views. A
  key aspect of our approach is the multi-task generation of images and depth
  maps, using learnable task embeddings to guide the diffusion process towards
  specific modalities. We train this model on a collection of more than 60
  million multi-view samples from publicly available datasets, and propose
  techniques to enable efficient and consistent learning in such diverse
  conditions. We also propose a novel strategy that enables the efficient
  training of larger models by incrementally fine-tuning smaller ones, with
  promising scaling behavior. Through extensive experiments, we report
  state-of-the-art results in multiple novel view synthesis benchmarks, as well
  as multi-view stereo and video depth estimation.
paper_authors._id: 67a10b8e83c3565727b0cd68
paper_authors.name: Vitor Guizilini
paper_authors._id: 67a10b8e83c3565727b0cd69
paper_authors.name: Muhammad Zubair Irshad
paper_authors._id: 67a10b8e83c3565727b0cd6a
paper_authors.name: Dian Chen
paper_authors._id: 67a10b8e83c3565727b0cd6b
paper_authors.name: Greg Shakhnarovich
paper_authors._id: 67a10b8e83c3565727b0cd6c
paper_authors.name: Rares Ambrus
paper_title: Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric
    Diffusion
paper_summary: Current methods for 3D scene reconstruction from sparse posed images employ
  intermediate 3D representations such as neural fields, voxel grids, or 3D
  Gaussians, to achieve multi-view consistent scene appearance and geometry. In
  this paper we introduce MVGD, a diffusion-based architecture capable of direct
  pixel-level generation of images and depth maps from novel viewpoints, given an
  arbitrary number of input views. Our method uses raymap conditioning to both
  augment visual features with spatial information from different viewpoints, as
  well as to guide the generation of images and depth maps from novel views. A
  key aspect of our approach is the multi-task generation of images and depth
  maps, using learnable task embeddings to guide the diffusion process towards
  specific modalities. We train this model on a collection of more than 60
  million multi-view samples from publicly available datasets, and propose
  techniques to enable efficient and consistent learning in such diverse
  conditions. We also propose a novel strategy that enables the efficient
  training of larger models by incrementally fine-tuning smaller ones, with
  promising scaling behavior. Through extensive experiments, we report
  state-of-the-art results in multiple novel view synthesis benchmarks, as well
  as multi-view stereo and video depth estimation.
paper_ai_summary: MVGD uses diffusion-based architecture with raymap conditioning to generate images and depth maps from novel viewpoints, achieving state-of-the-art results in novel view synthesis and depth estimation.
paper_ai_keywords: neural fields voxel grids 3D Gaussians MVGD diffusion-based architecture pixel-level generation raymap conditioning multi-task generation learnable task embeddings incremental fine-tuning novel view synthesis multi-view stereo video depth estimation