title: Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space
summary: Understanding the 3D semantics of a scene is a fundamental problem for
  various scenarios such as embodied agents. While NeRFs and 3DGS excel at
  novel-view synthesis, previous methods for understanding their semantics have
  been limited to incomplete 3D understanding: their segmentation results are 2D
  masks and their supervision is anchored at 2D pixels. This paper revisits the
  problem set to pursue a better 3D understanding of a scene modeled by NeRFs and
  3DGS as follows. 1) We directly supervise the 3D points to train the language
  embedding field. It achieves state-of-the-art accuracy without relying on
  multi-scale language embeddings. 2) We transfer the pre-trained language field
  to 3DGS, achieving the first real-time rendering speed without sacrificing
  training time or accuracy. 3) We introduce a 3D querying and evaluation
  protocol for assessing the reconstructed geometry and semantics together. Code,
  checkpoints, and annotations will be available online. Project page:
  https://hyunji12.github.io/Open3DRF
paper_authors._id: 66bd89f6dc1c1a06009bf02b
paper_authors.name: Hyunjee Lee
paper_authors._id: 66bd89f6dc1c1a06009bf02c
paper_authors.name: Youngsik Yun
paper_authors._id: 66bd89f6dc1c1a06009bf02d
paper_authors.name: Jeongmin Bae
paper_authors._id: 66bd89f6dc1c1a06009bf02e
paper_authors.name: Seoha Kim
paper_authors._id: 66bd89f6dc1c1a06009bf02f
paper_authors.name: Youngjung Uh
paper_title: Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space
paper_summary: Understanding the 3D semantics of a scene is a fundamental problem for
  various scenarios such as embodied agents. While NeRFs and 3DGS excel at
  novel-view synthesis, previous methods for understanding their semantics have
  been limited to incomplete 3D understanding: their segmentation results are 2D
  masks and their supervision is anchored at 2D pixels. This paper revisits the
  problem set to pursue a better 3D understanding of a scene modeled by NeRFs and
  3DGS as follows. 1) We directly supervise the 3D points to train the language
  embedding field. It achieves state-of-the-art accuracy without relying on
  multi-scale language embeddings. 2) We transfer the pre-trained language field
  to 3DGS, achieving the first real-time rendering speed without sacrificing
  training time or accuracy. 3) We introduce a 3D querying and evaluation
  protocol for assessing the reconstructed geometry and semantics together. Code,
  checkpoints, and annotations will be available online. Project page:
  https://hyunji12.github.io/Open3DRF
paper_ai_summary: The paper presents advancements in 3D scene understanding by directly supervising 3D points for language embedding fields, transferring these fields to 3DGS for real-time rendering, and introducing a 3D querying protocol for assessment.
paper_ai_keywords: NeRFs 3DGS 3D points language embedding field multi-scale language embeddings real-time rendering 3D querying 3D semantics scene understanding novel-view synthesis