QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds
<|reference_start|>QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds: While pets offer companionship, their limited intelligence restricts advanced reasoning and autonomous interaction with humans. Considering this, we propose QuadrupedGPT, a versatile agent designed to master a broad range of complex tasks with agility comparable to that of a pet. To achieve this goal, the primary challenges include: i) effectively leveraging multimodal observations for decision-making; ii) mastering agile control of locomotion and path planning; iii) developing advanced cognition to execute long-term objectives. QuadrupedGPT processes human command and environmental contexts using a large multimodal model (LMM). Empowered by its extensive knowledge base, our agent autonomously assigns appropriate parameters for adaptive locomotion policies and guides the agent in planning a safe but efficient path towards the goal, utilizing semantic-aware terrain analysis. Moreover, QuadrupedGPT is equipped with problem-solving capabilities that enable it to decompose long-term goals into a sequence of executable subgoals through high-level reasoning. Extensive experiments across various benchmarks confirm that QuadrupedGPT can adeptly handle multiple tasks with intricate instructions, demonstrating a significant step towards the versatile quadruped agents in open-ended worlds. Our website and codes can be found at https://quadruped-hub.github.io/Quadruped-GPT/.<|reference_end|>
@article{wang2024quadrupedgpt:,
    title={QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds},
    author={Ye Wang, Yuting Mei, Sipeng Zheng, Qin Jin},
    journal={arXiv preprint arXiv:2406.16578},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.16578},
    primaryClass={cs.RO cs.AI}
}