MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based
  Monocular Guidance
The latest regularized Neural Radiance Field (NeRF) approaches produce poor
geometry and view extrapolation for multiview stereo (MVS) benchmarks such as
ETH3D. In this paper, we aim to create 3D models that provide accurate geometry
and view synthesis, partially closing the large geometric performance gap
between NeRF and traditional MVS methods. We propose a patch-based approach
that effectively leverages monocular surface normal and relative depth
predictions. The patch-based ray sampling also enables the appearance
regularization of normalized cross-correlation (NCC) and structural similarity
(SSIM) between randomly sampled virtual and training views. We further show
that "density restrictions" based on sparse structure-from-motion points can
help greatly improve geometric accuracy with a slight drop in novel view
synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x
that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a
fruitful research direction to improve the geometric accuracy of NeRF-based
models, and sheds light on a potential future approach to enable NeRF-based
optimization to eventually outperform traditional MVS.
661c83c251c9d2b6b6e5fabe
Yuqun Wu
admin_assigned
2024-04-18T11:49:07.824Z
65766652b951d40e7a6ed846
/avatars/9a3e32e94bce1cd8253d67d438b377ca.svg
Yuqun Wu
user
yuqunw
661c83c251c9d2b6b6e5fabf
Jae Yong Lee
661c83c251c9d2b6b6e5fac0
Chuhang Zou
661c83c251c9d2b6b6e5fac1
Shenlong Wang
admin_assigned
2024-04-18T11:49:35.157Z
62fbe0aaa80632fbd47b86f5
/avatars/8175e3ac325745714f17dbe000ea955f.svg
Shenlong Wang
user
shenlong
661c83c251c9d2b6b6e5fac2
Derek Hoiem
MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based
  Monocular Guidance
The latest regularized Neural Radiance Field (NeRF) approaches produce poor
geometry and view extrapolation for multiview stereo (MVS) benchmarks such as
ETH3D. In this paper, we aim to create 3D models that provide accurate geometry
and view synthesis, partially closing the large geometric performance gap
between NeRF and traditional MVS methods. We propose a patch-based approach
that effectively leverages monocular surface normal and relative depth
predictions. The patch-based ray sampling also enables the appearance
regularization of normalized cross-correlation (NCC) and structural similarity
(SSIM) between randomly sampled virtual and training views. We further show
that "density restrictions" based on sparse structure-from-motion points can
help greatly improve geometric accuracy with a slight drop in novel view
synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x
that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a
fruitful research direction to improve the geometric accuracy of NeRF-based
models, and sheds light on a potential future approach to enable NeRF-based
optimization to eventually outperform traditional MVS.
A patch-based approach enhances the geometric accuracy and view synthesis of NeRFs by leveraging surface normals and depth predictions, showing significant performance improvements on MVS benchmarks.
Neural Radiance Field NeRF multiview stereo MVS ETH3D patch-based approach monocular surface normal relative depth normalized cross-correlation NCC structural similarity SSIM density restrictions structure-from-motion RegNeRF FreeNeRF average F1@2cm