ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields
  with Diffusion Models
We propose ExtraNeRF, a novel method for extrapolating the range of views
handled by a Neural Radiance Field (NeRF). Our main idea is to leverage NeRFs
to model scene-specific, fine-grained details, while capitalizing on diffusion
models to extrapolate beyond our observed data. A key ingredient is to track
visibility to determine what portions of the scene have not been observed, and
focus on reconstructing those regions consistently with diffusion models. Our
primary contributions include a visibility-aware diffusion-based inpainting
module that is fine-tuned on the input imagery, yielding an initial NeRF with
moderate quality (often blurry) inpainted regions, followed by a second
diffusion model trained on the input imagery to consistently enhance, notably
sharpen, the inpainted imagery from the first pass. We demonstrate high-quality
results, extrapolating beyond a small number of (typically six or fewer) input
views, effectively outpainting the NeRF as well as inpainting newly disoccluded
regions inside the original viewing volume. We compare with related work both
quantitatively and qualitatively and show significant gains over prior art.
6667d1ad0aa71e338ea5502f
Meng-Li Shih
admin_assigned
2024-06-11T08:38:37.674Z
64367b5c1adb261e94e2a940
/avatars/1774abe8f924a057f9a9ecf4cd6a7e3b.svg
Meng-Li Shih
user
MLShih
6667d1ad0aa71e338ea55030
Wei-Chiu Ma
admin_assigned
2024-06-11T08:39:09.693Z
624fb0c8841b0a7f1b583e72
/avatars/a695a528817f57a98c946056d6f9b2f9.svg
Wei-Chiu Ma
user
weichium
6667d1ad0aa71e338ea55031
Aleksander Holynski
admin_assigned
2024-06-11T08:39:03.460Z
62f6e2792e53c2efd33faa92
/avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
Aleksander Holynski
user
holynski
6667d1ad0aa71e338ea55032
Forrester Cole
admin_assigned
2024-06-11T08:38:56.865Z
635003011d81beb8e244e68d
/avatars/32f46e6778d1c52ebdd6f2511ba48257.svg
Forrester Cole
user
fcole
6667d1ad0aa71e338ea55033
Brian L. Curless
6667d1ad0aa71e338ea55034
Janne Kontkanen
ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields
  with Diffusion Models
We propose ExtraNeRF, a novel method for extrapolating the range of views
handled by a Neural Radiance Field (NeRF). Our main idea is to leverage NeRFs
to model scene-specific, fine-grained details, while capitalizing on diffusion
models to extrapolate beyond our observed data. A key ingredient is to track
visibility to determine what portions of the scene have not been observed, and
focus on reconstructing those regions consistently with diffusion models. Our
primary contributions include a visibility-aware diffusion-based inpainting
module that is fine-tuned on the input imagery, yielding an initial NeRF with
moderate quality (often blurry) inpainted regions, followed by a second
diffusion model trained on the input imagery to consistently enhance, notably
sharpen, the inpainted imagery from the first pass. We demonstrate high-quality
results, extrapolating beyond a small number of (typically six or fewer) input
views, effectively outpainting the NeRF as well as inpainting newly disoccluded
regions inside the original viewing volume. We compare with related work both
quantitatively and qualitatively and show significant gains over prior art.
ExtraNeRF combines NeRF and diffusion models to extrapolate NeRF views beyond observed data, using a visibility-aware inpainting module and subsequent quality enhancement.
Neural Radiance Fields NeRF diffusion models visibility-aware diffusion-based inpainting outpainting inpainting disoccluded regions