Solving Robotics Problems in Zero-Shot with Vision-Language Models
<|reference_start|>Solving Robotics Problems in Zero-Shot with Vision-Language Models: We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework designed to solve robotics problems in a zero-shot regime. In our context, zero-shot means that for a novel environment, we provide a VLLM with an image of the robot's surroundings and a task description, and the VLLM outputs the sequence of actions necessary for the robot to complete the task. Unlike prior work that requires fine-tuning parts of the pipeline -- such as adjusting an LLM on robot-specific data or training separate vision encoders -- our approach demonstrates that with careful engineering, a single off-the-shelf VLLM can autonomously handle all aspects of a robotics task, from high-level planning to low-level location extraction and action execution. Crucially, compared to using GPT-4o alone, Wonderful Team is self-corrective and capable of iteratively fixing its own mistakes, enabling it to solve challenging long-horizon tasks. We validate our framework through extensive experiments, both in simulated environments using VIMABench and in real-world settings. Our system showcases the ability to handle diverse tasks such as manipulation, goal-reaching, and visual reasoning -- all in a zero-shot manner. These results underscore a key point: vision-language models have progressed rapidly in the past year and should be strongly considered as a backbone for many robotics problems moving forward.<|reference_end|>
@article{wang2024solving,
    title={Solving Robotics Problems in Zero-Shot with Vision-Language Models},
    author={Zidan Wang, Rui Shen, Bradly Stadie},
    journal={arXiv preprint arXiv:2407.19094},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19094},
    primaryClass={cs.AI cs.RO}
}