LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
Large Language Models (LLMs) equipped with extensive world knowledge and
strong reasoning skills can tackle diverse tasks across domains, often by
posing them as conversation-style instruction-response pairs. In this paper, we
propose LLaRA: Large Language and Robotics Assistant, a framework which
formulates robot action policy as conversations, and provides improved
responses when trained with auxiliary data that complements policy learning.
LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity
to process state information as visual-textual prompts and generate optimal
policy decisions in text. To train such action policy VLMs, we first introduce
an automated pipeline to generate diverse high-quality robotics instruction
data from existing behavior cloning data. A VLM finetuned with the resulting
collection of datasets based on a conversation-style formulation tailored for
robotics tasks, can generate meaningful robot action policy decisions. Our
experiments across multiple simulated and real-world environments demonstrate
the state-of-the-art performance of the proposed LLaRA framework. The code,
datasets, and pretrained models are available at
https://github.com/LostXine/LLaRA.
668233727b50b433cdfc100a
Xiang Li
claimed_verified
2024-07-01T07:46:52.958Z
6369c6b17f319ce3573771ae
https://cdn-avatars.huggingface.co/v1/production/uploads/6369c6b17f319ce3573771ae/g7zVTXWBeeOBQqR5mOByS.jpeg
Xiang Li
user
variante
668233727b50b433cdfc100b
Cristina Mata
admin_assigned
2024-07-01T09:20:50.776Z
6462514fd1ccd517f46773e5
/avatars/e69ebb82df9f193db388d841d1fc996c.svg
Cristina Mata
user
cfmata
668233727b50b433cdfc100c
Jongwoo Park
admin_assigned
2024-07-01T09:21:47.581Z
666b7bf3f178c7e90115e038
/avatars/2a8f4cf26cdedb4fa5f715f0d14d1bbf.svg
Jongwoo Park
user
jongwoopark7978
668233727b50b433cdfc100d
Kumara Kahatapitiya
admin_assigned
2024-07-01T09:22:13.652Z
65a12ec4c9873890d15c4ab9
/avatars/ce4d46f575ba757f78eabdb25b394171.svg
Kumara Kahatapitiya
user
kumarak
668233727b50b433cdfc100e
Yoo Sung Jang
admin_assigned
2024-07-01T09:22:22.236Z
664775508fa42b4fe7f9638c
/avatars/302f0adca572a139023221ee6476e22f.svg
Yoo Sung Jang
user
yjang43
668233727b50b433cdfc100f
Jinghuan Shang
admin_assigned
2024-07-01T09:22:48.532Z
63f686c49cbd6730302cf0f2
/avatars/b553823be640a07eeb4c7edc2c176d5d.svg
Jinghuan Shang
user
Jinghuan
668233727b50b433cdfc1010
Kanchana Ranasinghe
admin_assigned
2024-07-01T09:22:55.586Z
64125da1227028b63e317ede
https://cdn-avatars.huggingface.co/v1/production/uploads/1678925215042-noauth.jpeg
Kanchana Ranasinghe
user
kahnchana
668233727b50b433cdfc1011
Ryan Burgert
admin_assigned
2024-07-01T09:23:01.806Z
62f69fe504e5e02f82aef7fb
https://cdn-avatars.huggingface.co/v1/production/uploads/62f69fe504e5e02f82aef7fb/m4CA0XlTKDGIg_WQVmGdv.png
Ryan Burgert
user
OneOverZero
668233727b50b433cdfc1012
Mu Cai
admin_assigned
2024-07-01T09:23:27.185Z
63b7b2c6bd2d153522821766
https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
Mu Cai
user
mucai
668233727b50b433cdfc1013
Yong Jae Lee
admin_assigned
2024-07-01T09:23:33.168Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
668233727b50b433cdfc1014
Michael S. Ryoo
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
Large Language Models (LLMs) equipped with extensive world knowledge and
strong reasoning skills can tackle diverse tasks across domains, often by
posing them as conversation-style instruction-response pairs. In this paper, we
propose LLaRA: Large Language and Robotics Assistant, a framework which
formulates robot action policy as conversations, and provides improved
responses when trained with auxiliary data that complements policy learning.
LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity
to process state information as visual-textual prompts and generate optimal
policy decisions in text. To train such action policy VLMs, we first introduce
an automated pipeline to generate diverse high-quality robotics instruction
data from existing behavior cloning data. A VLM finetuned with the resulting
collection of datasets based on a conversation-style formulation tailored for
robotics tasks, can generate meaningful robot action policy decisions. Our
experiments across multiple simulated and real-world environments demonstrate
the state-of-the-art performance of the proposed LLaRA framework. The code,
datasets, and pretrained models are available at
https://github.com/LostXine/LLaRA.
LLaRA, a framework for robotic action policies using conversation-style instruction, enhances performance through automatic data generation and fine-tuning of Vision Language Models.
Large Language Models LLaRA Robotics Assistant Vision Language Models behavior cloning automated pipeline conversation-style formulation simulated environments real-world environments