title: Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
    Primitives
summary: Given a set of calibrated images of a scene, we present an approach that
  produces a simple, compact, and actionable 3D world representation by means of
  3D primitives. While many approaches focus on recovering high-fidelity 3D
  scenes, we focus on parsing a scene into mid-level 3D representations made of a
  small set of textured primitives. Such representations are interpretable, easy
  to manipulate and suited for physics-based simulations. Moreover, unlike
  existing primitive decomposition methods that rely on 3D input data, our
  approach operates directly on images through differentiable rendering.
  Specifically, we model primitives as textured superquadric meshes and optimize
  their parameters from scratch with an image rendering loss. We highlight the
  importance of modeling transparency for each primitive, which is critical for
  optimization and also enables handling varying numbers of primitives. We show
  that the resulting textured primitives faithfully reconstruct the input images
  and accurately model the visible 3D points, while providing amodal shape
  completions of unseen object regions. We compare our approach to the state of
  the art on diverse scenes from DTU, and demonstrate its robustness on real-life
  captures from BlendedMVS and Nerfstudio. We also showcase how our results can
  be used to effortlessly edit a scene or perform physical simulations. Code and
  video results are available at https://www.tmonnier.com/DBW .
paper_authors._id: 64ae1d516e9a4384cfa10082
paper_authors.name: Tom Monnier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T13:28:44.641Z
paper_authors._id: 64ae7145324a758f7963eb19
paper_authors.avatarUrl: /avatars/d80e37b83ad298e37cd0496fa7557c7c.svg
paper_authors.fullname: Tom 
paper_authors.type: user
paper_authors.user: monniert
paper_authors._id: 64ae1d516e9a4384cfa10083
paper_authors.name: Jake Austin
paper_authors._id: 64ae1d516e9a4384cfa10084
paper_authors.name: Angjoo Kanazawa
paper_authors._id: 64ae1d516e9a4384cfa10085
paper_authors.name: Alexei A. Efros
paper_authors._id: 64ae1d516e9a4384cfa10086
paper_authors.name: Mathieu Aubry
paper_title: Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
    Primitives
paper_summary: Given a set of calibrated images of a scene, we present an approach that
  produces a simple, compact, and actionable 3D world representation by means of
  3D primitives. While many approaches focus on recovering high-fidelity 3D
  scenes, we focus on parsing a scene into mid-level 3D representations made of a
  small set of textured primitives. Such representations are interpretable, easy
  to manipulate and suited for physics-based simulations. Moreover, unlike
  existing primitive decomposition methods that rely on 3D input data, our
  approach operates directly on images through differentiable rendering.
  Specifically, we model primitives as textured superquadric meshes and optimize
  their parameters from scratch with an image rendering loss. We highlight the
  importance of modeling transparency for each primitive, which is critical for
  optimization and also enables handling varying numbers of primitives. We show
  that the resulting textured primitives faithfully reconstruct the input images
  and accurately model the visible 3D points, while providing amodal shape
  completions of unseen object regions. We compare our approach to the state of
  the art on diverse scenes from DTU, and demonstrate its robustness on real-life
  captures from BlendedMVS and Nerfstudio. We also showcase how our results can
  be used to effortlessly edit a scene or perform physical simulations. Code and
  video results are available at https://www.tmonnier.com/DBW .
paper_ai_keywords: 3D primitives differentiable rendering superquadric meshes image rendering loss transparency modeling amodal shape completions