Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
<|reference_start|>Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V: Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. We present COME-robot, the first closed-loop framework utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios. We meticulously construct a library of action primitives for robot exploration, navigation, and manipulation, serving as callable execution modules for GPT-4V in task planning. On top of these modules, GPT-4V serves as the brain that can accomplish multimodal reasoning, generate action policy with code, verify the task progress, and provide feedback for replanning. Such design enables COME-robot to (i) actively perceive the environments, (ii) perform situated reasoning, and (iii) recover from failures. Through comprehensive experiments involving 8 challenging real-world tabletop and manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~25%) compared to state-of-the-art baseline methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning.<|reference_end|>
@article{zhi2024closed-loop,
    title={Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V},
    author={Peiyuan Zhi, Zhiyuan Zhang, Muzhi Han, Zeyu Zhang, Zhitian Li, Ziyuan
  Jiao, Baoxiong Jia, Siyuan Huang},
    journal={arXiv preprint arXiv:2404.10220},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10220},
    primaryClass={cs.RO cs.AI cs.CV cs.LG}
}