Disentangled 3D Scene Generation with Layout Learning
We introduce a method to generate 3D scenes that are disentangled into their
component objects. This disentanglement is unsupervised, relying only on the
knowledge of a large pretrained text-to-image model. Our key insight is that
objects can be discovered by finding parts of a 3D scene that, when rearranged
spatially, still produce valid configurations of the same scene. Concretely,
our method jointly optimizes multiple NeRFs from scratch - each representing
its own object - along with a set of layouts that composite these objects into
scenes. We then encourage these composited scenes to be in-distribution
according to the image generator. We show that despite its simplicity, our
approach successfully generates 3D scenes decomposed into individual objects,
enabling new capabilities in text-to-3D content creation. For results and an
interactive demo, see our project page at https://dave.ml/layoutlearning/
65deb5036d161782405dd172
Dave Epstein
admin_assigned
2024-02-28T12:31:31.787Z
64caa93391726743360ab1f0
/avatars/992cb5aaf42d21be03a998656bb485d7.svg
Dave Epstein
user
daveepstein
65deb5036d161782405dd173
Ben Poole
admin_assigned
2024-02-28T12:31:37.752Z
63213768d2d45f315183abc3
/avatars/521f52f430b73e514afb06e3a9d6779d.svg
Ben Poole
user
doinkda
65deb5036d161782405dd174
Ben Mildenhall
65deb5036d161782405dd175
Alexei A. Efros
65deb5036d161782405dd176
Aleksander Holynski
admin_assigned
2024-02-28T12:31:57.221Z
62f6e2792e53c2efd33faa92
/avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
Aleksander Holynski
user
holynski
Disentangled 3D Scene Generation with Layout Learning
We introduce a method to generate 3D scenes that are disentangled into their
component objects. This disentanglement is unsupervised, relying only on the
knowledge of a large pretrained text-to-image model. Our key insight is that
objects can be discovered by finding parts of a 3D scene that, when rearranged
spatially, still produce valid configurations of the same scene. Concretely,
our method jointly optimizes multiple NeRFs from scratch - each representing
its own object - along with a set of layouts that composite these objects into
scenes. We then encourage these composited scenes to be in-distribution
according to the image generator. We show that despite its simplicity, our
approach successfully generates 3D scenes decomposed into individual objects,
enabling new capabilities in text-to-3D content creation. For results and an
interactive demo, see our project page at https://dave.ml/layoutlearning/
Unsupervised disentanglement of 3D scenes into component objects using pretrained text-to-image models and joint optimization of NeRFs and layouts.
NeRFs in-distribution text-to-3D scene decomposition object discovery