NeRF-Det: Learning Geometry-Aware Volumetric Representation for
  Multi-View 3D Object Detection
We present NeRF-Det, a novel method for indoor 3D detection with posed RGB
images as input. Unlike existing indoor 3D detection methods that struggle to
model scene geometry, our method makes novel use of NeRF in an end-to-end
manner to explicitly estimate 3D geometry, thereby improving 3D detection
performance. Specifically, to avoid the significant extra latency associated
with per-scene optimization of NeRF, we introduce sufficient geometry priors to
enhance the generalizability of NeRF-MLP. Furthermore, we subtly connect the
detection and NeRF branches through a shared MLP, enabling an efficient
adaptation of NeRF to detection and yielding geometry-aware volumetric
representations for 3D detection. Our method outperforms state-of-the-arts by
3.9 mAP and 3.1 mAP on the ScanNet and ARKITScenes benchmarks, respectively. We
provide extensive analysis to shed light on how NeRF-Det works. As a result of
our joint-training design, NeRF-Det is able to generalize well to unseen scenes
for object detection, view synthesis, and depth estimation tasks without
requiring per-scene optimization. Code is available at
https://github.com/facebookresearch/NeRF-Det.
64c3652c2a5eaefd000aed00
Chenfeng Xu
claimed_verified
2025-06-04T09:07:07.214Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
64c3652c2a5eaefd000aed01
Bichen Wu
admin_assigned
2023-07-28T10:02:11.618Z
6363749520c2cf37bccd9e84
/avatars/e9de7911e67090d883f15a7a58f18e4e.svg
Bichen Wu
user
bichenwu
64c3652c2a5eaefd000aed02
Ji Hou
admin_assigned
2023-07-28T10:02:45.982Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
64c3652c2a5eaefd000aed03
Sam Tsai
claimed_verified
2024-10-18T15:31:32.863Z
67127b753a058517519fabe8
/avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
Sam Tsai
user
sstsai-adl
64c3652c2a5eaefd000aed04
Ruilong Li
admin_assigned
2023-07-28T10:03:04.061Z
63adcd0d3e98f3ec193eea65
/avatars/8d85388c30c557638342787adf0fd4e4.svg
Ruilong LI
user
jccgyangtao
64c3652c2a5eaefd000aed05
Jialiang Wang
64c3652c2a5eaefd000aed06
Wei Zhan
64c3652c2a5eaefd000aed07
Zijian He
64c3652c2a5eaefd000aed08
Peter Vajda
admin_assigned
2023-07-28T10:03:38.665Z
636f4b35af79dd193603342b
/avatars/577328195738d92ac41a3d0ff3108d7c.svg
Peter Vajda
user
vaphab
64c3652c2a5eaefd000aed09
Kurt Keutzer
admin_assigned
2023-07-28T10:03:48.965Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
64c3652c2a5eaefd000aed0a
Masayoshi Tomizuka
NeRF-Det: Learning Geometry-Aware Volumetric Representation for
  Multi-View 3D Object Detection
We present NeRF-Det, a novel method for indoor 3D detection with posed RGB
images as input. Unlike existing indoor 3D detection methods that struggle to
model scene geometry, our method makes novel use of NeRF in an end-to-end
manner to explicitly estimate 3D geometry, thereby improving 3D detection
performance. Specifically, to avoid the significant extra latency associated
with per-scene optimization of NeRF, we introduce sufficient geometry priors to
enhance the generalizability of NeRF-MLP. Furthermore, we subtly connect the
detection and NeRF branches through a shared MLP, enabling an efficient
adaptation of NeRF to detection and yielding geometry-aware volumetric
representations for 3D detection. Our method outperforms state-of-the-arts by
3.9 mAP and 3.1 mAP on the ScanNet and ARKITScenes benchmarks, respectively. We
provide extensive analysis to shed light on how NeRF-Det works. As a result of
our joint-training design, NeRF-Det is able to generalize well to unseen scenes
for object detection, view synthesis, and depth estimation tasks without
requiring per-scene optimization. Code is available at
https://github.com/facebookresearch/NeRF-Det.
NeRF-Det uses NeRF for end-to-end 3D detection with posed RGB images, incorporating geometry priors and shared MLPs to improve performance and generalization.
NeRF NeRF-Det indoor 3D detection posed RGB images scene geometry NeRF-MLP shared MLP geometry-aware volumetric representations ScanNet ARKITScenes mAP