Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models
<|reference_start|>Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models: Recently, there has been considerable attention towards leveraging large language models (LLMs) to enhance decision-making processes. However, aligning the natural language text instructions generated by LLMs with the vectorized operations required for execution presents a significant challenge, often necessitating task-specific details. To circumvent the need for such task-specific granularity, inspired by preference-based policy learning approaches, we investigate the utilization of multimodal LLMs to provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks, serving as a critic to offer analysis and preference feedback. Subsequently, we validate the effectiveness of preference labels generated by CriticGPT from a reward modeling perspective. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Furthermore, performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models.<|reference_end|>
@article{liu2024enhancing,
    title={Enhancing Robotic Manipulation with AI Feedback from Multimodal Large
  Language Models},
    author={Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, Yan
  Zheng},
    journal={arXiv preprint arXiv:2402.14245},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.14245},
    primaryClass={cs.RO cs.AI cs.LG}
}