Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
  Primitives
Given a set of calibrated images of a scene, we present an approach that
produces a simple, compact, and actionable 3D world representation by means of
3D primitives. While many approaches focus on recovering high-fidelity 3D
scenes, we focus on parsing a scene into mid-level 3D representations made of a
small set of textured primitives. Such representations are interpretable, easy
to manipulate and suited for physics-based simulations. Moreover, unlike
existing primitive decomposition methods that rely on 3D input data, our
approach operates directly on images through differentiable rendering.
Specifically, we model primitives as textured superquadric meshes and optimize
their parameters from scratch with an image rendering loss. We highlight the
importance of modeling transparency for each primitive, which is critical for
optimization and also enables handling varying numbers of primitives. We show
that the resulting textured primitives faithfully reconstruct the input images
and accurately model the visible 3D points, while providing amodal shape
completions of unseen object regions. We compare our approach to the state of
the art on diverse scenes from DTU, and demonstrate its robustness on real-life
captures from BlendedMVS and Nerfstudio. We also showcase how our results can
be used to effortlessly edit a scene or perform physical simulations. Code and
video results are available at https://www.tmonnier.com/DBW .
64ae1d516e9a4384cfa10082
Tom Monnier
claimed_verified
2023-07-12T13:28:44.641Z
64ae7145324a758f7963eb19
/avatars/d80e37b83ad298e37cd0496fa7557c7c.svg
Tom 
user
monniert
64ae1d516e9a4384cfa10083
Jake Austin
64ae1d516e9a4384cfa10084
Angjoo Kanazawa
64ae1d516e9a4384cfa10085
Alexei A. Efros
64ae1d516e9a4384cfa10086
Mathieu Aubry
Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
  Primitives
Given a set of calibrated images of a scene, we present an approach that
produces a simple, compact, and actionable 3D world representation by means of
3D primitives. While many approaches focus on recovering high-fidelity 3D
scenes, we focus on parsing a scene into mid-level 3D representations made of a
small set of textured primitives. Such representations are interpretable, easy
to manipulate and suited for physics-based simulations. Moreover, unlike
existing primitive decomposition methods that rely on 3D input data, our
approach operates directly on images through differentiable rendering.
Specifically, we model primitives as textured superquadric meshes and optimize
their parameters from scratch with an image rendering loss. We highlight the
importance of modeling transparency for each primitive, which is critical for
optimization and also enables handling varying numbers of primitives. We show
that the resulting textured primitives faithfully reconstruct the input images
and accurately model the visible 3D points, while providing amodal shape
completions of unseen object regions. We compare our approach to the state of
the art on diverse scenes from DTU, and demonstrate its robustness on real-life
captures from BlendedMVS and Nerfstudio. We also showcase how our results can
be used to effortlessly edit a scene or perform physical simulations. Code and
video results are available at https://www.tmonnier.com/DBW .
The approach generates a compact 3D world representation using textured primitives, optimized through differentiable rendering, and performs well compared to existing methods, supporting scene editing and simulations.
3D primitives differentiable rendering textured superquadric meshes image rendering loss transparency modeling amodal shape completions DTU scenes BlendedMVS Nerfstudio scene editing physical simulations