title: Large Language Models as Generalizable Policies for Embodied Tasks
summary: We show that large language models (LLMs) can be adapted to be generalizable
  policies for embodied visual tasks. Our approach, called Large LAnguage model
  Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take
  as input text instructions and visual egocentric observations and output
  actions directly in the environment. Using reinforcement learning, we train
  LLaRP to see and act solely through environmental interactions. We show that
  LLaRP is robust to complex paraphrasings of task instructions and can
  generalize to new tasks that require novel optimal behavior. In particular, on
  1,000 unseen tasks it achieves 42% success rate, 1.7x the success rate of other
  common learned baselines or zero-shot applications of LLMs. Finally, to aid the
  community in studying language conditioned, massively multi-task, embodied AI
  problems we release a novel benchmark, Language Rearrangement, consisting of
  150,000 training and 1,000 testing tasks for language-conditioned
  rearrangement. Video examples of LLaRP in unseen Language Rearrangement
  instructions are at https://llm-rl.github.io.
paper_authors._id: 653f1125059fdd2587090c4f
paper_authors.name: Andrew Szot
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-30T15:08:49.745Z
paper_authors._id: 63ed2971c5b3c734085fd5ef
paper_authors.avatarUrl: /avatars/9bc509e4f1007c6087d67eff59d739dc.svg
paper_authors.fullname: Andrew
paper_authors.type: user
paper_authors.user: aszot
paper_authors._id: 653f1125059fdd2587090c50
paper_authors.name: Max Schwarzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T14:15:39.279Z
paper_authors._id: 6306f2de435ec751b730087e
paper_authors.avatarUrl: /avatars/7e2858cbc1526f6060a730f8ca3af290.svg
paper_authors.fullname: Max Schwarzer
paper_authors.type: user
paper_authors.user: MaxSchwarzer
paper_authors._id: 653f1125059fdd2587090c51
paper_authors.name: Harsh Agrawal
paper_authors._id: 653f1125059fdd2587090c52
paper_authors.name: Bogdan Mazoure
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T14:16:18.919Z
paper_authors._id: 630783068b054d85f2988b60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630783068b054d85f2988b60/-os9ZaVBpFU_Rqno9kPtR.jpeg
paper_authors.fullname: Bogdan Mazoure
paper_authors.type: user
paper_authors.user: bmazoure
paper_authors._id: 653f1125059fdd2587090c53
paper_authors.name: Walter Talbott
paper_authors._id: 653f1125059fdd2587090c54
paper_authors.name: Katherine Metcalf
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:25:27.821Z
paper_authors._id: 669968174bbe8ad52e889152
paper_authors.avatarUrl: /avatars/209eebe4eb16a5808cbf5abde9a051c3.svg
paper_authors.fullname: Rin Metcalf Susa
paper_authors.type: user
paper_authors.user: kmetcalf
paper_authors._id: 653f1125059fdd2587090c55
paper_authors.name: Natalie Mackraz
paper_authors._id: 653f1125059fdd2587090c56
paper_authors.name: Devon Hjelm
paper_authors._id: 653f1125059fdd2587090c57
paper_authors.name: Alexander Toshev
paper_title: Large Language Models as Generalizable Policies for Embodied Tasks
paper_summary: We show that large language models (LLMs) can be adapted to be generalizable
  policies for embodied visual tasks. Our approach, called Large LAnguage model
  Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take
  as input text instructions and visual egocentric observations and output
  actions directly in the environment. Using reinforcement learning, we train
  LLaRP to see and act solely through environmental interactions. We show that
  LLaRP is robust to complex paraphrasings of task instructions and can
  generalize to new tasks that require novel optimal behavior. In particular, on
  1,000 unseen tasks it achieves 42% success rate, 1.7x the success rate of other
  common learned baselines or zero-shot applications of LLMs. Finally, to aid the
  community in studying language conditioned, massively multi-task, embodied AI
  problems we release a novel benchmark, Language Rearrangement, consisting of
  150,000 training and 1,000 testing tasks for language-conditioned
  rearrangement. Video examples of LLaRP in unseen Language Rearrangement
  instructions are at https://llm-rl.github.io.
paper_ai_keywords: Large language models (LLMs) LLaRP (Large LAnguage model Reinforcement Learning Policy) reinforcement learning text instructions visual egocentric observations generalizable policies embodied visual tasks complex paraphrasings novel optimal behavior language conditioned massively multi-task embodied AI Language Rearrangement language-conditioned rearrangement