title: Plan, Eliminate, and Track -- Language Models are Good Teachers for
    Embodied Agents
summary: Pre-trained large language models (LLMs) capture procedural knowledge about
  the world. Recent work has leveraged LLM's ability to generate abstract plans
  to simplify challenging control tasks, either by action scoring, or action
  modeling (fine-tuning). However, the transformer architecture inherits several
  constraints that make it difficult for the LLM to directly serve as the agent:
  e.g. limited input lengths, fine-tuning inefficiency, bias from pre-training,
  and incompatibility with non-text environments. To maintain compatibility with
  a low-level trainable actor, we propose to instead use the knowledge in LLMs to
  simplify the control problem, rather than solving it. We propose the Plan,
  Eliminate, and Track (PET) framework. The Plan module translates a task
  description into a list of high-level sub-tasks. The Eliminate module masks out
  irrelevant objects and receptacles from the observation for the current
  sub-task. Finally, the Track module determines whether the agent has
  accomplished each sub-task. On the AlfWorld instruction following benchmark,
  the PET framework leads to a significant 15% improvement over SOTA for
  generalization to human goal specifications.
paper_authors._id: 645477c9afea4a7004a1c531
paper_authors.name: Yue Wu
paper_authors._id: 645477c9afea4a7004a1c532
paper_authors.name: So Yeon Min
paper_authors._id: 645477c9afea4a7004a1c533
paper_authors.name: Yonatan Bisk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:25:38.204Z
paper_authors._id: 63180f3c879073cf83f135b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
paper_authors.fullname: Yonatan Bisk
paper_authors.type: user
paper_authors.user: ybisk
paper_authors._id: 645477c9afea4a7004a1c534
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 645477c9afea4a7004a1c535
paper_authors.name: Amos Azaria
paper_authors._id: 645477c9afea4a7004a1c536
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:26:24.610Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_authors._id: 645477c9afea4a7004a1c537
paper_authors.name: Tom Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:26:52.862Z
paper_authors._id: 6439ac853ab54fdbab81c5c4
paper_authors.avatarUrl: /avatars/5592fb0420e393a821ae60e6280e1553.svg
paper_authors.fullname: Tom Mitchell
paper_authors.type: user
paper_authors.user: tommmitchell
paper_authors._id: 645477c9afea4a7004a1c538
paper_authors.name: Shrimai Prabhumoye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:27:13.134Z
paper_authors._id: 6470efe7fa9fd77212e17a97
paper_authors.avatarUrl: /avatars/a116b279b927627eeee9c23021b6e5ee.svg
paper_authors.fullname: Shrimai Prabhumoye
paper_authors.type: user
paper_authors.user: shrimai
paper_title: Plan, Eliminate, and Track -- Language Models are Good Teachers for
    Embodied Agents
paper_summary: Pre-trained large language models (LLMs) capture procedural knowledge about
  the world. Recent work has leveraged LLM's ability to generate abstract plans
  to simplify challenging control tasks, either by action scoring, or action
  modeling (fine-tuning). However, the transformer architecture inherits several
  constraints that make it difficult for the LLM to directly serve as the agent:
  e.g. limited input lengths, fine-tuning inefficiency, bias from pre-training,
  and incompatibility with non-text environments. To maintain compatibility with
  a low-level trainable actor, we propose to instead use the knowledge in LLMs to
  simplify the control problem, rather than solving it. We propose the Plan,
  Eliminate, and Track (PET) framework. The Plan module translates a task
  description into a list of high-level sub-tasks. The Eliminate module masks out
  irrelevant objects and receptacles from the observation for the current
  sub-task. Finally, the Track module determines whether the agent has
  accomplished each sub-task. On the AlfWorld instruction following benchmark,
  the PET framework leads to a significant 15% improvement over SOTA for
  generalization to human goal specifications.
paper_ai_keywords: probabilistic large language models (LLMs) transformer architecture input lengths fine-tuning inefficiency bias from pre-training incompatibility with non-text environments Plan module high-level sub-tasks Eliminate module Track module AlfWorld instruction following benchmark generalization to human goal specifications