title: LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
summary: Large Language Models (LLMs) equipped with extensive world knowledge and
  strong reasoning skills can tackle diverse tasks across domains, often by
  posing them as conversation-style instruction-response pairs. In this paper, we
  propose LLaRA: Large Language and Robotics Assistant, a framework which
  formulates robot action policy as conversations, and provides improved
  responses when trained with auxiliary data that complements policy learning.
  LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity
  to process state information as visual-textual prompts and generate optimal
  policy decisions in text. To train such action policy VLMs, we first introduce
  an automated pipeline to generate diverse high-quality robotics instruction
  data from existing behavior cloning data. A VLM finetuned with the resulting
  collection of datasets based on a conversation-style formulation tailored for
  robotics tasks, can generate meaningful robot action policy decisions. Our
  experiments across multiple simulated and real-world environments demonstrate
  the state-of-the-art performance of the proposed LLaRA framework. The code,
  datasets, and pretrained models are available at
  https://github.com/LostXine/LLaRA.
paper_authors._id: 668233727b50b433cdfc100a
paper_authors.name: Xiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-01T07:46:52.958Z
paper_authors._id: 6369c6b17f319ce3573771ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6369c6b17f319ce3573771ae/g7zVTXWBeeOBQqR5mOByS.jpeg
paper_authors.fullname: Xiang Li
paper_authors.type: user
paper_authors.user: variante
paper_authors._id: 668233727b50b433cdfc100b
paper_authors.name: Cristina Mata
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:20:50.776Z
paper_authors._id: 6462514fd1ccd517f46773e5
paper_authors.avatarUrl: /avatars/e69ebb82df9f193db388d841d1fc996c.svg
paper_authors.fullname: Cristina Mata
paper_authors.type: user
paper_authors.user: cfmata
paper_authors._id: 668233727b50b433cdfc100c
paper_authors.name: Jongwoo Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:21:47.581Z
paper_authors._id: 666b7bf3f178c7e90115e038
paper_authors.avatarUrl: /avatars/2a8f4cf26cdedb4fa5f715f0d14d1bbf.svg
paper_authors.fullname: Jongwoo Park
paper_authors.type: user
paper_authors.user: jongwoopark7978
paper_authors._id: 668233727b50b433cdfc100d
paper_authors.name: Kumara Kahatapitiya
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:22:13.652Z
paper_authors._id: 65a12ec4c9873890d15c4ab9
paper_authors.avatarUrl: /avatars/ce4d46f575ba757f78eabdb25b394171.svg
paper_authors.fullname: Kumara Kahatapitiya
paper_authors.type: user
paper_authors.user: kumarak
paper_authors._id: 668233727b50b433cdfc100e
paper_authors.name: Yoo Sung Jang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:22:22.236Z
paper_authors._id: 664775508fa42b4fe7f9638c
paper_authors.avatarUrl: /avatars/302f0adca572a139023221ee6476e22f.svg
paper_authors.fullname: Yoo Sung Jang
paper_authors.type: user
paper_authors.user: yjang43
paper_authors._id: 668233727b50b433cdfc100f
paper_authors.name: Jinghuan Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:22:48.532Z
paper_authors._id: 63f686c49cbd6730302cf0f2
paper_authors.avatarUrl: /avatars/b553823be640a07eeb4c7edc2c176d5d.svg
paper_authors.fullname: Jinghuan Shang
paper_authors.type: user
paper_authors.user: Jinghuan
paper_authors._id: 668233727b50b433cdfc1010
paper_authors.name: Kanchana Ranasinghe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:22:55.586Z
paper_authors._id: 64125da1227028b63e317ede
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678925215042-noauth.jpeg
paper_authors.fullname: Kanchana Ranasinghe
paper_authors.type: user
paper_authors.user: kahnchana
paper_authors._id: 668233727b50b433cdfc1011
paper_authors.name: Ryan Burgert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:23:01.806Z
paper_authors._id: 62f69fe504e5e02f82aef7fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f69fe504e5e02f82aef7fb/m4CA0XlTKDGIg_WQVmGdv.png
paper_authors.fullname: Ryan Burgert
paper_authors.type: user
paper_authors.user: OneOverZero
paper_authors._id: 668233727b50b433cdfc1012
paper_authors.name: Mu Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:23:27.185Z
paper_authors._id: 63b7b2c6bd2d153522821766
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
paper_authors.fullname: Mu Cai
paper_authors.type: user
paper_authors.user: mucai
paper_authors._id: 668233727b50b433cdfc1013
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:23:33.168Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_authors._id: 668233727b50b433cdfc1014
paper_authors.name: Michael S. Ryoo
paper_title: LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
paper_summary: Large Language Models (LLMs) equipped with extensive world knowledge and
  strong reasoning skills can tackle diverse tasks across domains, often by
  posing them as conversation-style instruction-response pairs. In this paper, we
  propose LLaRA: Large Language and Robotics Assistant, a framework which
  formulates robot action policy as conversations, and provides improved
  responses when trained with auxiliary data that complements policy learning.
  LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity
  to process state information as visual-textual prompts and generate optimal
  policy decisions in text. To train such action policy VLMs, we first introduce
  an automated pipeline to generate diverse high-quality robotics instruction
  data from existing behavior cloning data. A VLM finetuned with the resulting
  collection of datasets based on a conversation-style formulation tailored for
  robotics tasks, can generate meaningful robot action policy decisions. Our
  experiments across multiple simulated and real-world environments demonstrate
  the state-of-the-art performance of the proposed LLaRA framework. The code,
  datasets, and pretrained models are available at
  https://github.com/LostXine/LLaRA.
paper_ai_summary: LLaRA, a framework for robotic action policies using conversation-style instruction, enhances performance through automatic data generation and fine-tuning of Vision Language Models.
paper_ai_keywords: Large Language Models LLaRA Robotics Assistant Vision Language Models behavior cloning automated pipeline conversation-style formulation simulated environments real-world environments