title: MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based
    Monocular Guidance
summary: The latest regularized Neural Radiance Field (NeRF) approaches produce poor
  geometry and view extrapolation for multiview stereo (MVS) benchmarks such as
  ETH3D. In this paper, we aim to create 3D models that provide accurate geometry
  and view synthesis, partially closing the large geometric performance gap
  between NeRF and traditional MVS methods. We propose a patch-based approach
  that effectively leverages monocular surface normal and relative depth
  predictions. The patch-based ray sampling also enables the appearance
  regularization of normalized cross-correlation (NCC) and structural similarity
  (SSIM) between randomly sampled virtual and training views. We further show
  that "density restrictions" based on sparse structure-from-motion points can
  help greatly improve geometric accuracy with a slight drop in novel view
  synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x
  that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a
  fruitful research direction to improve the geometric accuracy of NeRF-based
  models, and sheds light on a potential future approach to enable NeRF-based
  optimization to eventually outperform traditional MVS.
paper_authors._id: 661c83c251c9d2b6b6e5fabe
paper_authors.name: Yuqun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:49:07.824Z
paper_authors._id: 65766652b951d40e7a6ed846
paper_authors.avatarUrl: /avatars/9a3e32e94bce1cd8253d67d438b377ca.svg
paper_authors.fullname: Yuqun Wu
paper_authors.type: user
paper_authors.user: yuqunw
paper_authors._id: 661c83c251c9d2b6b6e5fabf
paper_authors.name: Jae Yong Lee
paper_authors._id: 661c83c251c9d2b6b6e5fac0
paper_authors.name: Chuhang Zou
paper_authors._id: 661c83c251c9d2b6b6e5fac1
paper_authors.name: Shenlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:49:35.157Z
paper_authors._id: 62fbe0aaa80632fbd47b86f5
paper_authors.avatarUrl: /avatars/8175e3ac325745714f17dbe000ea955f.svg
paper_authors.fullname: Shenlong Wang
paper_authors.type: user
paper_authors.user: shenlong
paper_authors._id: 661c83c251c9d2b6b6e5fac2
paper_authors.name: Derek Hoiem
paper_title: MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based
    Monocular Guidance
paper_summary: The latest regularized Neural Radiance Field (NeRF) approaches produce poor
  geometry and view extrapolation for multiview stereo (MVS) benchmarks such as
  ETH3D. In this paper, we aim to create 3D models that provide accurate geometry
  and view synthesis, partially closing the large geometric performance gap
  between NeRF and traditional MVS methods. We propose a patch-based approach
  that effectively leverages monocular surface normal and relative depth
  predictions. The patch-based ray sampling also enables the appearance
  regularization of normalized cross-correlation (NCC) and structural similarity
  (SSIM) between randomly sampled virtual and training views. We further show
  that "density restrictions" based on sparse structure-from-motion points can
  help greatly improve geometric accuracy with a slight drop in novel view
  synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x
  that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a
  fruitful research direction to improve the geometric accuracy of NeRF-based
  models, and sheds light on a potential future approach to enable NeRF-based
  optimization to eventually outperform traditional MVS.
paper_ai_keywords: Neural Radiance Field (NeRF) multiview stereo (MVS) ETH3D patch-based approach monocular surface normal relative depth ray sampling normalized cross-correlation (NCC) structural similarity (SSIM) virtual views training views density restrictions sparse structure-from-motion points RegNeRF FreeNeRF F1@2cm ETH3D MVS benchmark geometric accuracy novel view synthesis