title: Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting
summary: 3D scene generation has quickly become a challenging new research direction,
  fueled by consistent improvements of 2D generative diffusion models. Most prior
  work in this area generates scenes by iteratively stitching newly generated
  frames with existing geometry. These works often depend on pre-trained
  monocular depth estimators to lift the generated images into 3D, fusing them
  with the existing scene representation. These approaches are then often
  evaluated via a text metric, measuring the similarity between the generated
  images and a given text prompt. In this work, we make two fundamental
  contributions to the field of 3D scene generation. First, we note that lifting
  images to 3D with a monocular depth estimation model is suboptimal as it
  ignores the geometry of the existing scene. We thus introduce a novel depth
  completion model, trained via teacher distillation and self-training to learn
  the 3D fusion process, resulting in improved geometric coherence of the scene.
  Second, we introduce a new benchmarking scheme for scene generation methods
  that is based on ground truth geometry, and thus measures the quality of the
  structure of the scene.
paper_authors._id: 6631c6de6aee5b05e4a0b14d
paper_authors.name: Paul Engstler
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:10:55.882Z
paper_authors._id: 62262603a8218a9b862b18cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648321672399-62262603a8218a9b862b18cc.jpeg
paper_authors.fullname: Paul Engstler
paper_authors.type: user
paper_authors.user: paulengstler
paper_authors._id: 6631c6de6aee5b05e4a0b14e
paper_authors.name: Andrea Vedaldi
paper_authors._id: 6631c6de6aee5b05e4a0b14f
paper_authors.name: Iro Laina
paper_authors._id: 6631c6de6aee5b05e4a0b150
paper_authors.name: Christian Rupprecht
paper_title: Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting
paper_summary: 3D scene generation has quickly become a challenging new research direction,
  fueled by consistent improvements of 2D generative diffusion models. Most prior
  work in this area generates scenes by iteratively stitching newly generated
  frames with existing geometry. These works often depend on pre-trained
  monocular depth estimators to lift the generated images into 3D, fusing them
  with the existing scene representation. These approaches are then often
  evaluated via a text metric, measuring the similarity between the generated
  images and a given text prompt. In this work, we make two fundamental
  contributions to the field of 3D scene generation. First, we note that lifting
  images to 3D with a monocular depth estimation model is suboptimal as it
  ignores the geometry of the existing scene. We thus introduce a novel depth
  completion model, trained via teacher distillation and self-training to learn
  the 3D fusion process, resulting in improved geometric coherence of the scene.
  Second, we introduce a new benchmarking scheme for scene generation methods
  that is based on ground truth geometry, and thus measures the quality of the
  structure of the scene.
paper_ai_keywords: generative diffusion models monocular depth estimators text metric depth detecting model teacher distillation self-training geometric coherence ground truth geometry