title: NeRF-Det: Learning Geometry-Aware Volumetric Representation for
    Multi-View 3D Object Detection
summary: We present NeRF-Det, a novel method for indoor 3D detection with posed RGB
  images as input. Unlike existing indoor 3D detection methods that struggle to
  model scene geometry, our method makes novel use of NeRF in an end-to-end
  manner to explicitly estimate 3D geometry, thereby improving 3D detection
  performance. Specifically, to avoid the significant extra latency associated
  with per-scene optimization of NeRF, we introduce sufficient geometry priors to
  enhance the generalizability of NeRF-MLP. Furthermore, we subtly connect the
  detection and NeRF branches through a shared MLP, enabling an efficient
  adaptation of NeRF to detection and yielding geometry-aware volumetric
  representations for 3D detection. Our method outperforms state-of-the-arts by
  3.9 mAP and 3.1 mAP on the ScanNet and ARKITScenes benchmarks, respectively. We
  provide extensive analysis to shed light on how NeRF-Det works. As a result of
  our joint-training design, NeRF-Det is able to generalize well to unseen scenes
  for object detection, view synthesis, and depth estimation tasks without
  requiring per-scene optimization. Code is available at
  https://github.com/facebookresearch/NeRF-Det.
paper_authors._id: 64c3652c2a5eaefd000aed00
paper_authors.name: Chenfeng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:07:07.214Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_authors._id: 64c3652c2a5eaefd000aed01
paper_authors.name: Bichen Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T10:02:11.618Z
paper_authors._id: 6363749520c2cf37bccd9e84
paper_authors.avatarUrl: /avatars/e9de7911e67090d883f15a7a58f18e4e.svg
paper_authors.fullname: Bichen Wu
paper_authors.type: user
paper_authors.user: bichenwu
paper_authors._id: 64c3652c2a5eaefd000aed02
paper_authors.name: Ji Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T10:02:45.982Z
paper_authors._id: 63084636f0dc38fb47c2dc6b
paper_authors.avatarUrl: /avatars/04edff10a048b5d93981e6f5af3ecf40.svg
paper_authors.fullname: Ji Hou
paper_authors.type: user
paper_authors.user: jhou90
paper_authors._id: 64c3652c2a5eaefd000aed03
paper_authors.name: Sam Tsai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T15:31:32.863Z
paper_authors._id: 67127b753a058517519fabe8
paper_authors.avatarUrl: /avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
paper_authors.fullname: Sam Tsai
paper_authors.type: user
paper_authors.user: sstsai-adl
paper_authors._id: 64c3652c2a5eaefd000aed04
paper_authors.name: Ruilong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T10:03:04.061Z
paper_authors._id: 63adcd0d3e98f3ec193eea65
paper_authors.avatarUrl: /avatars/8d85388c30c557638342787adf0fd4e4.svg
paper_authors.fullname: Ruilong LI
paper_authors.type: user
paper_authors.user: jccgyangtao
paper_authors._id: 64c3652c2a5eaefd000aed05
paper_authors.name: Jialiang Wang
paper_authors._id: 64c3652c2a5eaefd000aed06
paper_authors.name: Wei Zhan
paper_authors._id: 64c3652c2a5eaefd000aed07
paper_authors.name: Zijian He
paper_authors._id: 64c3652c2a5eaefd000aed08
paper_authors.name: Peter Vajda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T10:03:38.665Z
paper_authors._id: 636f4b35af79dd193603342b
paper_authors.avatarUrl: /avatars/577328195738d92ac41a3d0ff3108d7c.svg
paper_authors.fullname: Peter Vajda
paper_authors.type: user
paper_authors.user: vaphab
paper_authors._id: 64c3652c2a5eaefd000aed09
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T10:03:48.965Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 64c3652c2a5eaefd000aed0a
paper_authors.name: Masayoshi Tomizuka
paper_title: NeRF-Det: Learning Geometry-Aware Volumetric Representation for
    Multi-View 3D Object Detection
paper_summary: We present NeRF-Det, a novel method for indoor 3D detection with posed RGB
  images as input. Unlike existing indoor 3D detection methods that struggle to
  model scene geometry, our method makes novel use of NeRF in an end-to-end
  manner to explicitly estimate 3D geometry, thereby improving 3D detection
  performance. Specifically, to avoid the significant extra latency associated
  with per-scene optimization of NeRF, we introduce sufficient geometry priors to
  enhance the generalizability of NeRF-MLP. Furthermore, we subtly connect the
  detection and NeRF branches through a shared MLP, enabling an efficient
  adaptation of NeRF to detection and yielding geometry-aware volumetric
  representations for 3D detection. Our method outperforms state-of-the-arts by
  3.9 mAP and 3.1 mAP on the ScanNet and ARKITScenes benchmarks, respectively. We
  provide extensive analysis to shed light on how NeRF-Det works. As a result of
  our joint-training design, NeRF-Det is able to generalize well to unseen scenes
  for object detection, view synthesis, and depth estimation tasks without
  requiring per-scene optimization. Code is available at
  https://github.com/facebookresearch/NeRF-Det.
paper_ai_summary: NeRF-Det uses NeRF for end-to-end 3D detection with posed RGB images, incorporating geometry priors and shared MLPs to improve performance and generalization.
paper_ai_keywords: NeRF NeRF-Det indoor 3D detection posed RGB images scene geometry NeRF-MLP shared MLP geometry-aware volumetric representations ScanNet ARKITScenes mAP