3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and
  Box-Focused Sampling for 3D Object Detection
Neural Radiance Fields (NeRF) are widely used for novel-view synthesis and
have been adapted for 3D Object Detection (3DOD), offering a promising approach
to 3DOD through view-synthesis representation. However, NeRF faces inherent
limitations: (i) limited representational capacity for 3DOD due to its implicit
nature, and (ii) slow rendering speeds. Recently, 3D Gaussian Splatting (3DGS)
has emerged as an explicit 3D representation that addresses these limitations.
Inspired by these advantages, this paper introduces 3DGS into 3DOD for the
first time, identifying two main challenges: (i) Ambiguous spatial distribution
of Gaussian blobs: 3DGS primarily relies on 2D pixel-level supervision,
resulting in unclear 3D spatial distribution of Gaussian blobs and poor
differentiation between objects and background, which hinders 3DOD; (ii)
Excessive background blobs: 2D images often include numerous background pixels,
leading to densely reconstructed 3DGS with many noisy Gaussian blobs
representing the background, negatively affecting detection. To tackle the
challenge (i), we leverage the fact that 3DGS reconstruction is derived from 2D
images, and propose an elegant and efficient solution by incorporating 2D
Boundary Guidance to significantly enhance the spatial distribution of Gaussian
blobs, resulting in clearer differentiation between objects and their
background. To address the challenge (ii), we propose a Box-Focused Sampling
strategy using 2D boxes to generate object probability distribution in 3D
spaces, allowing effective probabilistic sampling in 3D to retain more object
blobs and reduce noisy background blobs. Benefiting from our designs, our
3DGS-DET significantly outperforms the SOTA NeRF-based method, NeRF-Det,
achieving improvements of +6.6 on mAP@0.25 and +8.1 on mAP@0.5 for the ScanNet
dataset, and impressive +31.5 on mAP@0.25 for the ARKITScenes dataset.
66fe459a609fa71aa6f715e7
Yang Cao
admin_assigned
2024-10-03T08:58:43.253Z
647d893bcfca67bc50f8974f
/avatars/fab275f968b7d648dbdf8485c9279fb6.svg
Yang Cao
user
YangCaoCS
66fe459a609fa71aa6f715e8
Yuanliang Jv
admin_assigned
2024-10-03T08:58:10.852Z
646f4cf46b3df773a2c93fd0
https://cdn-avatars.huggingface.co/v1/production/uploads/646f4cf46b3df773a2c93fd0/8SGp5O6tmbd26Oxte9Gpy.jpeg
Yuanliang Ju
user
Yuanliang
66fe459a609fa71aa6f715e9
Dan Xu
claimed_verified
2024-10-11T16:33:14.238Z
66feab48651e00e22f33222e
/avatars/7344377e2c796c7ec85194bb2fc78521.svg
Dan Xu
user
danxuhk
3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and
  Box-Focused Sampling for 3D Object Detection
Neural Radiance Fields (NeRF) are widely used for novel-view synthesis and
have been adapted for 3D Object Detection (3DOD), offering a promising approach
to 3DOD through view-synthesis representation. However, NeRF faces inherent
limitations: (i) limited representational capacity for 3DOD due to its implicit
nature, and (ii) slow rendering speeds. Recently, 3D Gaussian Splatting (3DGS)
has emerged as an explicit 3D representation that addresses these limitations.
Inspired by these advantages, this paper introduces 3DGS into 3DOD for the
first time, identifying two main challenges: (i) Ambiguous spatial distribution
of Gaussian blobs: 3DGS primarily relies on 2D pixel-level supervision,
resulting in unclear 3D spatial distribution of Gaussian blobs and poor
differentiation between objects and background, which hinders 3DOD; (ii)
Excessive background blobs: 2D images often include numerous background pixels,
leading to densely reconstructed 3DGS with many noisy Gaussian blobs
representing the background, negatively affecting detection. To tackle the
challenge (i), we leverage the fact that 3DGS reconstruction is derived from 2D
images, and propose an elegant and efficient solution by incorporating 2D
Boundary Guidance to significantly enhance the spatial distribution of Gaussian
blobs, resulting in clearer differentiation between objects and their
background. To address the challenge (ii), we propose a Box-Focused Sampling
strategy using 2D boxes to generate object probability distribution in 3D
spaces, allowing effective probabilistic sampling in 3D to retain more object
blobs and reduce noisy background blobs. Benefiting from our designs, our
3DGS-DET significantly outperforms the SOTA NeRF-based method, NeRF-Det,
achieving improvements of +6.6 on mAP@0.25 and +8.1 on mAP@0.5 for the ScanNet
dataset, and impressive +31.5 on mAP@0.25 for the ARKITScenes dataset.
3D Gaussian Splatting is introduced for 3D object detection, improving NeRF's limitations through 2D Boundary Guidance and Box-Focused Sampling strategies, achieving superior performance on ScanNet and ARKITScenes datasets.
Neural Radiance Fields 3D Object Detection view-synthesis representation implicit nature rendering speeds 3D Gaussian Splatting explicit 3D representation Gaussian blobs 2D pixel-level supervision 2D Boundary Guidance Box-Focused Sampling object probability distribution mAP@0.25 mAP@0.5 ScanNet ARKITScenes