Where to Fetch: Extracting Visual Scene Representation from Large Pre-Trained Models for Robotic Goal Navigation
<|reference_start|>Where to Fetch: Extracting Visual Scene Representation from Large Pre-Trained Models for Robotic Goal Navigation: To complete a complex task where a robot navigates to a goal object and fetches it, the robot needs to have a good understanding of the instructions and the surrounding environment. Large pre-trained models have shown capabilities to interpret tasks defined via language descriptions. However, previous methods attempting to integrate large pre-trained models with daily tasks are not competent in many robotic goal navigation tasks due to poor understanding of the environment. In this work, we present a visual scene representation built with large-scale visual language models to form a feature representation of the environment capable of handling natural language queries. Combined with large language models, this method can parse language instructions into action sequences for a robot to follow, and accomplish goal navigation with querying the scene representation. Experiments demonstrate that our method enables the robot to follow a wide range of instructions and complete complex goal navigation tasks.<|reference_end|>
@article{li2024where,
    title={Where to Fetch: Extracting Visual Scene Representation from Large
  Pre-Trained Models for Robotic Goal Navigation},
    author={Yu Li, Dayou Li, Chenkun Zhao, Ruifeng Wang, Ran Song and Wei Zhang},
    journal={arXiv preprint arXiv:2408.10578},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.10578},
    primaryClass={cs.RO}
}