Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space
Understanding the 3D semantics of a scene is a fundamental problem for
various scenarios such as embodied agents. While NeRFs and 3DGS excel at
novel-view synthesis, previous methods for understanding their semantics have
been limited to incomplete 3D understanding: their segmentation results are 2D
masks and their supervision is anchored at 2D pixels. This paper revisits the
problem set to pursue a better 3D understanding of a scene modeled by NeRFs and
3DGS as follows. 1) We directly supervise the 3D points to train the language
embedding field. It achieves state-of-the-art accuracy without relying on
multi-scale language embeddings. 2) We transfer the pre-trained language field
to 3DGS, achieving the first real-time rendering speed without sacrificing
training time or accuracy. 3) We introduce a 3D querying and evaluation
protocol for assessing the reconstructed geometry and semantics together. Code,
checkpoints, and annotations will be available online. Project page:
https://hyunji12.github.io/Open3DRF
66bd89f6dc1c1a06009bf02b
Hyunjee Lee
66bd89f6dc1c1a06009bf02c
Youngsik Yun
66bd89f6dc1c1a06009bf02d
Jeongmin Bae
66bd89f6dc1c1a06009bf02e
Seoha Kim
66bd89f6dc1c1a06009bf02f
Youngjung Uh
Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space
Understanding the 3D semantics of a scene is a fundamental problem for
various scenarios such as embodied agents. While NeRFs and 3DGS excel at
novel-view synthesis, previous methods for understanding their semantics have
been limited to incomplete 3D understanding: their segmentation results are 2D
masks and their supervision is anchored at 2D pixels. This paper revisits the
problem set to pursue a better 3D understanding of a scene modeled by NeRFs and
3DGS as follows. 1) We directly supervise the 3D points to train the language
embedding field. It achieves state-of-the-art accuracy without relying on
multi-scale language embeddings. 2) We transfer the pre-trained language field
to 3DGS, achieving the first real-time rendering speed without sacrificing
training time or accuracy. 3) We introduce a 3D querying and evaluation
protocol for assessing the reconstructed geometry and semantics together. Code,
checkpoints, and annotations will be available online. Project page:
https://hyunji12.github.io/Open3DRF
The paper presents advancements in 3D scene understanding by directly supervising 3D points for language embedding fields, transferring these fields to 3DGS for real-time rendering, and introducing a 3D querying protocol for assessment.
NeRFs 3DGS 3D points language embedding field multi-scale language embeddings real-time rendering 3D querying 3D semantics scene understanding novel-view synthesis