NeRFiller: Completing Scenes via Generative 3D Inpainting
We propose NeRFiller, an approach that completes missing portions of a 3D
capture via generative 3D inpainting using off-the-shelf 2D visual generative
models. Often parts of a captured 3D scene or object are missing due to mesh
reconstruction failures or a lack of observations (e.g., contact regions, such
as the bottom of objects, or hard-to-reach areas). We approach this challenging
3D inpainting problem by leveraging a 2D inpainting diffusion model. We
identify a surprising behavior of these models, where they generate more 3D
consistent inpaints when images form a 2times2 grid, and show how to
generalize this behavior to more than four images. We then present an iterative
framework to distill these inpainted regions into a single consistent 3D scene.
In contrast to related works, we focus on completing scenes rather than
deleting foreground objects, and our approach does not require tight 2D object
masks or text. We compare our approach to relevant baselines adapted to our
setting on a variety of scenes, where NeRFiller creates the most 3D consistent
and plausible scene completions. Our project page is at
https://ethanweber.me/nerfiller.
65729a6d3ada38603f9b222d
Ethan Weber
admin_assigned
2023-12-08T10:42:16.118Z
6378485d9e1c10ecc27dc48e
https://cdn-avatars.huggingface.co/v1/production/uploads/6378485d9e1c10ecc27dc48e/Wnv495k1VdGyAuDvgiSeP.png
Ethan Weber
user
ethanweber
65729a6d3ada38603f9b222e
Aleksander Hołyński
65729a6d3ada38603f9b222f
Varun Jampani
admin_assigned
2023-12-08T10:42:26.622Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
65729a6d3ada38603f9b2230
Saurabh Saxena
admin_assigned
2023-12-08T10:42:43.741Z
6480af48de559d48dbb20d95
/avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
Saurabh Saxena
user
saurabhsaxena
65729a6d3ada38603f9b2231
Noah Snavely
admin_assigned
2023-12-08T10:42:50.941Z
632b79ca0a9dc02985af9e0b
/avatars/e2ca585398a3212b2956a14804d0ba67.svg
Noah Snavely
user
jimantha
65729a6d3ada38603f9b2232
Abhishek Kar
claimed_verified
2023-12-11T08:16:35.451Z
65730b1fa5f2213c55dae1db
/avatars/cd3bec5588e406bf0c8ba53705ed6123.svg
Abhishek Kar
user
akar43
65729a6d3ada38603f9b2233
Angjoo Kanazawa
admin_assigned
2023-12-08T10:43:16.301Z
6478cf7150ff7001631679c3
/avatars/65ec385a9cc44c972e6caf952e759ff1.svg
Angjoo Kanazawa
user
akanazawa
NeRFiller: Completing Scenes via Generative 3D Inpainting
We propose NeRFiller, an approach that completes missing portions of a 3D
capture via generative 3D inpainting using off-the-shelf 2D visual generative
models. Often parts of a captured 3D scene or object are missing due to mesh
reconstruction failures or a lack of observations (e.g., contact regions, such
as the bottom of objects, or hard-to-reach areas). We approach this challenging
3D inpainting problem by leveraging a 2D inpainting diffusion model. We
identify a surprising behavior of these models, where they generate more 3D
consistent inpaints when images form a 2times2 grid, and show how to
generalize this behavior to more than four images. We then present an iterative
framework to distill these inpainted regions into a single consistent 3D scene.
In contrast to related works, we focus on completing scenes rather than
deleting foreground objects, and our approach does not require tight 2D object
masks or text. We compare our approach to relevant baselines adapted to our
setting on a variety of scenes, where NeRFiller creates the most 3D consistent
and plausible scene completions. Our project page is at
https://ethanweber.me/nerfiller.
NeRFiller utilizes a 2D inpainting diffusion model to generate consistent 3D scene completions from sparse observations.
generative 3D inpainting diffusion models 2D inpainting diffusion model scene completion 3D scene scene reconstruction