title: ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields
    with Diffusion Models
summary: We propose ExtraNeRF, a novel method for extrapolating the range of views
  handled by a Neural Radiance Field (NeRF). Our main idea is to leverage NeRFs
  to model scene-specific, fine-grained details, while capitalizing on diffusion
  models to extrapolate beyond our observed data. A key ingredient is to track
  visibility to determine what portions of the scene have not been observed, and
  focus on reconstructing those regions consistently with diffusion models. Our
  primary contributions include a visibility-aware diffusion-based inpainting
  module that is fine-tuned on the input imagery, yielding an initial NeRF with
  moderate quality (often blurry) inpainted regions, followed by a second
  diffusion model trained on the input imagery to consistently enhance, notably
  sharpen, the inpainted imagery from the first pass. We demonstrate high-quality
  results, extrapolating beyond a small number of (typically six or fewer) input
  views, effectively outpainting the NeRF as well as inpainting newly disoccluded
  regions inside the original viewing volume. We compare with related work both
  quantitatively and qualitatively and show significant gains over prior art.
paper_authors._id: 6667d1ad0aa71e338ea5502f
paper_authors.name: Meng-Li Shih
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:38:37.674Z
paper_authors._id: 64367b5c1adb261e94e2a940
paper_authors.avatarUrl: /avatars/1774abe8f924a057f9a9ecf4cd6a7e3b.svg
paper_authors.fullname: Meng-Li Shih
paper_authors.type: user
paper_authors.user: MLShih
paper_authors._id: 6667d1ad0aa71e338ea55030
paper_authors.name: Wei-Chiu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:39:09.693Z
paper_authors._id: 624fb0c8841b0a7f1b583e72
paper_authors.avatarUrl: /avatars/a695a528817f57a98c946056d6f9b2f9.svg
paper_authors.fullname: Wei-Chiu Ma
paper_authors.type: user
paper_authors.user: weichium
paper_authors._id: 6667d1ad0aa71e338ea55031
paper_authors.name: Aleksander Holynski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:39:03.460Z
paper_authors._id: 62f6e2792e53c2efd33faa92
paper_authors.avatarUrl: /avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
paper_authors.fullname: Aleksander Holynski
paper_authors.type: user
paper_authors.user: holynski
paper_authors._id: 6667d1ad0aa71e338ea55032
paper_authors.name: Forrester Cole
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T08:38:56.865Z
paper_authors._id: 635003011d81beb8e244e68d
paper_authors.avatarUrl: /avatars/32f46e6778d1c52ebdd6f2511ba48257.svg
paper_authors.fullname: Forrester Cole
paper_authors.type: user
paper_authors.user: fcole
paper_authors._id: 6667d1ad0aa71e338ea55033
paper_authors.name: Brian L. Curless
paper_authors._id: 6667d1ad0aa71e338ea55034
paper_authors.name: Janne Kontkanen
paper_title: ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields
    with Diffusion Models
paper_summary: We propose ExtraNeRF, a novel method for extrapolating the range of views
  handled by a Neural Radiance Field (NeRF). Our main idea is to leverage NeRFs
  to model scene-specific, fine-grained details, while capitalizing on diffusion
  models to extrapolate beyond our observed data. A key ingredient is to track
  visibility to determine what portions of the scene have not been observed, and
  focus on reconstructing those regions consistently with diffusion models. Our
  primary contributions include a visibility-aware diffusion-based inpainting
  module that is fine-tuned on the input imagery, yielding an initial NeRF with
  moderate quality (often blurry) inpainted regions, followed by a second
  diffusion model trained on the input imagery to consistently enhance, notably
  sharpen, the inpainted imagery from the first pass. We demonstrate high-quality
  results, extrapolating beyond a small number of (typically six or fewer) input
  views, effectively outpainting the NeRF as well as inpainting newly disoccluded
  regions inside the original viewing volume. We compare with related work both
  quantitatively and qualitatively and show significant gains over prior art.
paper_ai_summary: ExtraNeRF combines NeRF and diffusion models to extrapolate NeRF views beyond observed data, using a visibility-aware inpainting module and subsequent quality enhancement.
paper_ai_keywords: Neural Radiance Fields NeRF diffusion models visibility-aware diffusion-based inpainting outpainting inpainting disoccluded regions