title: Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models
summary: We present a latent diffusion model over 3D scenes, that can be trained using
  only 2D image data. To achieve this, we first design an autoencoder that maps
  multi-view images to 3D Gaussian splats, and simultaneously builds a compressed
  latent representation of these splats. Then, we train a multi-view diffusion
  model over the latent space to learn an efficient generative model. This
  pipeline does not require object masks nor depths, and is suitable for complex
  scenes with arbitrary camera positions. We conduct careful experiments on two
  large-scale datasets of complex real-world scenes -- MVImgNet and
  RealEstate10K. We show that our approach enables generating 3D scenes in as
  little as 0.2 seconds, either from scratch, from a single input view, or from
  sparse input views. It produces diverse and high-quality results while running
  an order of magnitude faster than non-latent diffusion models and earlier
  NeRF-based generative models
paper_authors._id: 667609215eaa9dd299f4feee
paper_authors.name: Paul Henderson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:30:00.485Z
paper_authors._id: 66749366b82e48404d157a66
paper_authors.avatarUrl: /avatars/a56066447b0ebcc26c31ed2388a08707.svg
paper_authors.fullname: Paul Henderson
paper_authors.type: user
paper_authors.user: pmh47
paper_authors._id: 667609215eaa9dd299f4feef
paper_authors.name: Melonie de Almeida
paper_authors._id: 667609215eaa9dd299f4fef0
paper_authors.name: Daniela Ivanova
paper_authors._id: 667609215eaa9dd299f4fef1
paper_authors.name: Titas Anciukevičius
paper_title: Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models
paper_summary: We present a latent diffusion model over 3D scenes, that can be trained using
  only 2D image data. To achieve this, we first design an autoencoder that maps
  multi-view images to 3D Gaussian splats, and simultaneously builds a compressed
  latent representation of these splats. Then, we train a multi-view diffusion
  model over the latent space to learn an efficient generative model. This
  pipeline does not require object masks nor depths, and is suitable for complex
  scenes with arbitrary camera positions. We conduct careful experiments on two
  large-scale datasets of complex real-world scenes -- MVImgNet and
  RealEstate10K. We show that our approach enables generating 3D scenes in as
  little as 0.2 seconds, either from scratch, from a single input view, or from
  sparse input views. It produces diverse and high-quality results while running
  an order of magnitude faster than non-latent diffusion models and earlier
  NeRF-based generative models
paper_ai_summary: A latent diffusion model capable of generating 3D scenes from 2D images enables fast and high-quality scene synthesis without masks or depth data.
paper_ai_keywords: latent diffusion model multi-view images 3D Gaussian splats compressed latent representation multi-view diffusion model latent space MVImgNet RealEstate10K scene synthesis generative model NeRF-based generative models