Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation
<|reference_start|>Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation: Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.<|reference_end|>
@article{werby2024hierarchical,
    title={Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot
  Navigation},
    author={Abdelrhman Werby, Chenguang Huang, Martin B"uchner, Abhinav Valada,
  Wolfram Burgard},
    journal={arXiv preprint arXiv:2403.17846},
    year={2024},
    doi={10.15607/RSS.2024.XX.077},
    archivePrefix={arXiv},
    eprint={2403.17846},
    primaryClass={cs.RO cs.AI cs.CL cs.CV cs.LG}
}