title: NeRFiller: Completing Scenes via Generative 3D Inpainting
summary: We propose NeRFiller, an approach that completes missing portions of a 3D
  capture via generative 3D inpainting using off-the-shelf 2D visual generative
  models. Often parts of a captured 3D scene or object are missing due to mesh
  reconstruction failures or a lack of observations (e.g., contact regions, such
  as the bottom of objects, or hard-to-reach areas). We approach this challenging
  3D inpainting problem by leveraging a 2D inpainting diffusion model. We
  identify a surprising behavior of these models, where they generate more 3D
  consistent inpaints when images form a 2times2 grid, and show how to
  generalize this behavior to more than four images. We then present an iterative
  framework to distill these inpainted regions into a single consistent 3D scene.
  In contrast to related works, we focus on completing scenes rather than
  deleting foreground objects, and our approach does not require tight 2D object
  masks or text. We compare our approach to relevant baselines adapted to our
  setting on a variety of scenes, where NeRFiller creates the most 3D consistent
  and plausible scene completions. Our project page is at
  https://ethanweber.me/nerfiller.
paper_authors._id: 65729a6d3ada38603f9b222d
paper_authors.name: Ethan Weber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:42:16.118Z
paper_authors._id: 6378485d9e1c10ecc27dc48e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6378485d9e1c10ecc27dc48e/Wnv495k1VdGyAuDvgiSeP.png
paper_authors.fullname: Ethan Weber
paper_authors.type: user
paper_authors.user: ethanweber
paper_authors._id: 65729a6d3ada38603f9b222e
paper_authors.name: Aleksander Hołyński
paper_authors._id: 65729a6d3ada38603f9b222f
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:42:26.622Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_authors._id: 65729a6d3ada38603f9b2230
paper_authors.name: Saurabh Saxena
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:42:43.741Z
paper_authors._id: 6480af48de559d48dbb20d95
paper_authors.avatarUrl: /avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
paper_authors.fullname: Saurabh Saxena
paper_authors.type: user
paper_authors.user: saurabhsaxena
paper_authors._id: 65729a6d3ada38603f9b2231
paper_authors.name: Noah Snavely
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:42:50.941Z
paper_authors._id: 632b79ca0a9dc02985af9e0b
paper_authors.avatarUrl: /avatars/e2ca585398a3212b2956a14804d0ba67.svg
paper_authors.fullname: Noah Snavely
paper_authors.type: user
paper_authors.user: jimantha
paper_authors._id: 65729a6d3ada38603f9b2232
paper_authors.name: Abhishek Kar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-11T08:16:35.451Z
paper_authors._id: 65730b1fa5f2213c55dae1db
paper_authors.avatarUrl: /avatars/cd3bec5588e406bf0c8ba53705ed6123.svg
paper_authors.fullname: Abhishek Kar
paper_authors.type: user
paper_authors.user: akar43
paper_authors._id: 65729a6d3ada38603f9b2233
paper_authors.name: Angjoo Kanazawa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:43:16.301Z
paper_authors._id: 6478cf7150ff7001631679c3
paper_authors.avatarUrl: /avatars/65ec385a9cc44c972e6caf952e759ff1.svg
paper_authors.fullname: Angjoo Kanazawa
paper_authors.type: user
paper_authors.user: akanazawa
paper_title: NeRFiller: Completing Scenes via Generative 3D Inpainting
paper_summary: We propose NeRFiller, an approach that completes missing portions of a 3D
  capture via generative 3D inpainting using off-the-shelf 2D visual generative
  models. Often parts of a captured 3D scene or object are missing due to mesh
  reconstruction failures or a lack of observations (e.g., contact regions, such
  as the bottom of objects, or hard-to-reach areas). We approach this challenging
  3D inpainting problem by leveraging a 2D inpainting diffusion model. We
  identify a surprising behavior of these models, where they generate more 3D
  consistent inpaints when images form a 2times2 grid, and show how to
  generalize this behavior to more than four images. We then present an iterative
  framework to distill these inpainted regions into a single consistent 3D scene.
  In contrast to related works, we focus on completing scenes rather than
  deleting foreground objects, and our approach does not require tight 2D object
  masks or text. We compare our approach to relevant baselines adapted to our
  setting on a variety of scenes, where NeRFiller creates the most 3D consistent
  and plausible scene completions. Our project page is at
  https://ethanweber.me/nerfiller.
paper_ai_summary: NeRFiller utilizes a 2D inpainting diffusion model to generate consistent 3D scene completions from sparse observations.
paper_ai_keywords: generative 3D inpainting diffusion models 2D inpainting diffusion model scene completion 3D scene scene reconstruction