DORSal: Diffusion for Object-centric Representations of Scenes
  et al.
Recent progress in 3D scene understanding enables scalable learning of
representations across large datasets of diverse scenes. As a consequence,
generalization to unseen scenes and objects, rendering novel views from just a
single or a handful of input images, and controllable scene generation that
supports editing, is now possible. However, training jointly on a large number
of scenes typically compromises rendering quality when compared to single-scene
optimized models such as NeRFs. In this paper, we leverage recent progress in
diffusion models to equip 3D scene representation learning models with the
ability to render high-fidelity novel views, while retaining benefits such as
object-level scene editing to a large degree. In particular, we propose DORSal,
which adapts a video diffusion architecture for 3D scene generation conditioned
on object-centric slot-based representations of scenes. On both complex
synthetic multi-object scenes and on the real-world large-scale Street View
dataset, we show that DORSal enables scalable neural rendering of 3D scenes
with object-level editing and improves upon existing approaches.
648bf1bc335645aa03f03965
Allan Jabri
648bf1bc335645aa03f03966
Sjoerd van Steenkiste
648bf1bc335645aa03f03967
Emiel Hoogeboom
648bf1bc335645aa03f03968
Mehdi S. M. Sajjadi
648bf1bc335645aa03f03969
Thomas Kipf
claimed_verified
2023-06-16T10:52:05.911Z
6413572b6cd62eb3ba2024e9
https://cdn-avatars.huggingface.co/v1/production/uploads/6413572b6cd62eb3ba2024e9/UCtMdeV8_N6w-SBnTDXBy.jpeg
Thomas Kipf
user
tkipf
DORSal: Diffusion for Object-centric Representations of Scenes
  et al.
Recent progress in 3D scene understanding enables scalable learning of
representations across large datasets of diverse scenes. As a consequence,
generalization to unseen scenes and objects, rendering novel views from just a
single or a handful of input images, and controllable scene generation that
supports editing, is now possible. However, training jointly on a large number
of scenes typically compromises rendering quality when compared to single-scene
optimized models such as NeRFs. In this paper, we leverage recent progress in
diffusion models to equip 3D scene representation learning models with the
ability to render high-fidelity novel views, while retaining benefits such as
object-level scene editing to a large degree. In particular, we propose DORSal,
which adapts a video diffusion architecture for 3D scene generation conditioned
on object-centric slot-based representations of scenes. On both complex
synthetic multi-object scenes and on the real-world large-scale Street View
dataset, we show that DORSal enables scalable neural rendering of 3D scenes
with object-level editing and improves upon existing approaches.
A video diffusion architecture adapted for 3D scene generation improves rendering quality and scalability while maintaining object-level editing capabilities.
diffusion models DORSal video diffusion architecture 3D scene generation object-centric slot-based representations neural rendering