Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph
<|reference_start|>Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph: Locating objects described in natural language presents a significant challenge for autonomous agents. Existing CLIP-based open-vocabulary methods successfully perform 3D object grounding with simple (bare) queries, but cannot cope with ambiguous descriptions that demand an understanding of object relations. To tackle this problem, we propose a modular approach called BBQ (Beyond Bare Queries), which constructs 3D scene graph representation with metric and semantic edges and utilizes a large language model as a human-to-agent interface through our deductive scene reasoning algorithm. BBQ employs robust DINO-powered associations to construct 3D object-centric map and an advanced raycasting algorithm with a 2D vision-language model to describe them as graph nodes. On the Replica and ScanNet datasets, we have demonstrated that BBQ takes a leading place in open-vocabulary 3D semantic segmentation compared to other zero-shot methods. Also, we show that leveraging spatial relations is especially effective for scenes containing multiple entities of the same semantic class. On challenging Sr3D+, Nr3D and ScanRefer benchmarks, our deductive approach demonstrates a significant improvement, enabling objects grounding by complex queries compared to other state-of-the-art methods. The combination of our design choices and software implementation has resulted in significant data processing speed in experiments on the robot on-board computer. This promising performance enables the application of our approach in intelligent robotics projects. We made the code publicly available at https://linukc.github.io/BeyondBareQueries/.<|reference_end|>
@article{linok2024beyond,
    title={Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene
  Graph},
    author={Sergey Linok, Tatiana Zemskova, Svetlana Ladanova, Roman Titkov,
  Dmitry Yudin, Maxim Monastyrny, Aleksei Valenkov},
    journal={arXiv preprint arXiv:2406.07113},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.07113},
    primaryClass={cs.CV cs.AI}
}