Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting
3D scene generation has quickly become a challenging new research direction,
fueled by consistent improvements of 2D generative diffusion models. Most prior
work in this area generates scenes by iteratively stitching newly generated
frames with existing geometry. These works often depend on pre-trained
monocular depth estimators to lift the generated images into 3D, fusing them
with the existing scene representation. These approaches are then often
evaluated via a text metric, measuring the similarity between the generated
images and a given text prompt. In this work, we make two fundamental
contributions to the field of 3D scene generation. First, we note that lifting
images to 3D with a monocular depth estimation model is suboptimal as it
ignores the geometry of the existing scene. We thus introduce a novel depth
completion model, trained via teacher distillation and self-training to learn
the 3D fusion process, resulting in improved geometric coherence of the scene.
Second, we introduce a new benchmarking scheme for scene generation methods
that is based on ground truth geometry, and thus measures the quality of the
structure of the scene.
6631c6de6aee5b05e4a0b14d
Paul Engstler
claimed_verified
2024-05-02T07:10:55.882Z
62262603a8218a9b862b18cc
https://cdn-avatars.huggingface.co/v1/production/uploads/1648321672399-62262603a8218a9b862b18cc.jpeg
Paul Engstler
user
paulengstler
6631c6de6aee5b05e4a0b14e
Andrea Vedaldi
6631c6de6aee5b05e4a0b14f
Iro Laina
6631c6de6aee5b05e4a0b150
Christian Rupprecht
Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting
3D scene generation has quickly become a challenging new research direction,
fueled by consistent improvements of 2D generative diffusion models. Most prior
work in this area generates scenes by iteratively stitching newly generated
frames with existing geometry. These works often depend on pre-trained
monocular depth estimators to lift the generated images into 3D, fusing them
with the existing scene representation. These approaches are then often
evaluated via a text metric, measuring the similarity between the generated
images and a given text prompt. In this work, we make two fundamental
contributions to the field of 3D scene generation. First, we note that lifting
images to 3D with a monocular depth estimation model is suboptimal as it
ignores the geometry of the existing scene. We thus introduce a novel depth
completion model, trained via teacher distillation and self-training to learn
the 3D fusion process, resulting in improved geometric coherence of the scene.
Second, we introduce a new benchmarking scheme for scene generation methods
that is based on ground truth geometry, and thus measures the quality of the
structure of the scene.
Proposes a new depth completion model and benchmarking scheme for 3D scene generation, improving geometric coherence and structural quality.
generative diffusion models monocular depth estimators scene generation depth completion model teacher distillation self-training ground truth geometry