Search3D: Hierarchical Open-Vocabulary 3D Segmentation
<|reference_start|>Search3D: Hierarchical Open-Vocabulary 3D Segmentation: Open-vocabulary 3D segmentation enables the exploration of 3D spaces using free-form text descriptions. Existing methods for open-vocabulary 3D instance segmentation primarily focus on identifying object-level instances in a scene. However, they face challenges when it comes to understanding more fine-grained scene entities such as object parts, or regions described by generic attributes. In this work, we introduce Search3D, an approach that builds a hierarchical open-vocabulary 3D scene representation, enabling the search for entities at varying levels of granularity: fine-grained object parts, entire objects, or regions described by attributes like materials. Our method aims to expand the capabilities of open vocabulary instance-level 3D segmentation by shifting towards a more flexible open-vocabulary 3D search setting less anchored to explicit object-centric queries, compared to prior work. To ensure a systematic evaluation, we also contribute a scene-scale open-vocabulary 3D part segmentation benchmark based on MultiScan, along with a set of open-vocabulary fine-grained part annotations on ScanNet++. We verify the effectiveness of Search3D across several tasks, demonstrating that our approach outperforms baselines in scene-scale open-vocabulary 3D part segmentation, while maintaining strong performance in segmenting 3D objects and materials.<|reference_end|>
@article{takmaz2024search3d:,
    title={Search3D: Hierarchical Open-Vocabulary 3D Segmentation},
    author={Ayca Takmaz, Alexandros Delitzas, Robert W. Sumner, Francis Engelmann,
  Johanna Wald, Federico Tombari},
    journal={arXiv preprint arXiv:2409.18431},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.18431},
    primaryClass={cs.CV}
}