OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding
<|reference_start|>OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding: This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding. Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D point-level tasks due to weak feature expressiveness and inaccurate 2D-3D feature associations. To ensure robust feature presentation and 3D point-level understanding, we first employ SAM masks without cross-frame associations to train instance features with 3D consistency. These features exhibit both intra-object consistency and inter-object distinction. Then, we propose a two-stage codebook to discretize these features from coarse to fine levels. At the coarse level, we consider the positional information of 3D points to achieve location-based clustering, which is then refined at the fine level. Finally, we introduce an instance-level 3D-2D feature association method that links 3D points to 2D masks, which are further associated with 2D CLIP features. Extensive experiments, including open vocabulary-based 3D object selection, 3D point cloud understanding, click-based 3D object selection, and ablation studies, demonstrate the effectiveness of our proposed method. Project page: https://3d-aigc.github.io/OpenGaussian<|reference_end|>
@article{wu2024opengaussian:,
    title={OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary
  Understanding},
    author={Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua
  Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, Jian Zhang},
    journal={arXiv preprint arXiv:2406.02058},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02058},
    primaryClass={cs.CV cs.RO}
}