Online Embedding Multi-Scale CLIP Features into 3D Maps
<|reference_start|>Online Embedding Multi-Scale CLIP Features into 3D Maps: This study introduces a novel approach to online embedding of multi-scale CLIP (Contrastive Language-Image Pre-Training) features into 3D maps. By harnessing CLIP, this methodology surpasses the constraints of conventional vocabulary-limited methods and enables the incorporation of semantic information into the resultant maps. While recent approaches have explored the embedding of multi-modal features in maps, they often impose significant computational costs, lacking practicality for exploring unfamiliar environments in real time. Our approach tackles these challenges by efficiently computing and embedding multi-scale CLIP features, thereby facilitating the exploration of unfamiliar environments through real-time map generation. Moreover, the embedding CLIP features into the resultant maps makes offline retrieval via linguistic queries feasible. In essence, our approach simultaneously achieves real-time object search and mapping of unfamiliar environments. Additionally, we propose a zero-shot object-goal navigation system based on our mapping approach, and we validate its efficacy through object-goal navigation, offline object retrieval, and multi-object-goal navigation in both simulated environments and real robot experiments. The findings demonstrate that our method not only exhibits swifter performance than state-of-the-art mapping methods but also surpasses them in terms of the success rate of object-goal navigation tasks.<|reference_end|>
@article{taguchi2024online,
    title={Online Embedding Multi-Scale CLIP Features into 3D Maps},
    author={Shun Taguchi and Hideki Deguchi},
    journal={arXiv preprint arXiv:2403.18178},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.18178},
    primaryClass={cs.RO cs.CV}
}