title: Embodied Task Planning with Large Language Models
summary: Equipping embodied agents with commonsense is important for robots to
  successfully complete complex human instructions in general environments.
  Recent large language models (LLM) can embed rich semantic knowledge for agents
  in plan generation of complex tasks, while they lack the information about the
  realistic world and usually yield infeasible action sequences. In this paper,
  we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
  with physical scene constraint, where the agent generates executable plans
  according to the existed objects in the scene by aligning LLMs with the visual
  perception models. Specifically, we first construct a multimodal dataset
  containing triplets of indoor scenes, instructions and action plans, where we
  provide the designed prompts and the list of existing objects in the scene for
  GPT-3.5 to generate a large number of instructions and corresponding planned
  actions. The generated data is leveraged for grounded plan tuning of
  pre-trained LLMs. During inference, we discover the objects in the scene by
  extending open-vocabulary object detectors to multi-view RGB images collected
  in different achievable locations. Experimental results show that the generated
  plan from our TaPA framework can achieve higher success rate than LLaVA and
  GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
  planning in general and complex environments.
paper_authors._id: 64a62489f9a6bc9db6b17d0a
paper_authors.name: Zhenyu Wu
paper_authors._id: 64a62489f9a6bc9db6b17d0b
paper_authors.name: Ziwei Wang
paper_authors._id: 64a62489f9a6bc9db6b17d0c
paper_authors.name: Xiuwei Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:24:59.569Z
paper_authors._id: 648ac65fd044b25978015634
paper_authors.avatarUrl: /avatars/2278a66fdc953220e9f8fc0ccce3ff00.svg
paper_authors.fullname: Xiuwei Xu
paper_authors.type: user
paper_authors.user: xuxw98
paper_authors._id: 64a62489f9a6bc9db6b17d0d
paper_authors.name: Jiwen Lu
paper_authors._id: 64a62489f9a6bc9db6b17d0e
paper_authors.name: Haibin Yan
paper_title: Embodied Task Planning with Large Language Models
paper_summary: Equipping embodied agents with commonsense is important for robots to
  successfully complete complex human instructions in general environments.
  Recent large language models (LLM) can embed rich semantic knowledge for agents
  in plan generation of complex tasks, while they lack the information about the
  realistic world and usually yield infeasible action sequences. In this paper,
  we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
  with physical scene constraint, where the agent generates executable plans
  according to the existed objects in the scene by aligning LLMs with the visual
  perception models. Specifically, we first construct a multimodal dataset
  containing triplets of indoor scenes, instructions and action plans, where we
  provide the designed prompts and the list of existing objects in the scene for
  GPT-3.5 to generate a large number of instructions and corresponding planned
  actions. The generated data is leveraged for grounded plan tuning of
  pre-trained LLMs. During inference, we discover the objects in the scene by
  extending open-vocabulary object detectors to multi-view RGB images collected
  in different achievable locations. Experimental results show that the generated
  plan from our TaPA framework can achieve higher success rate than LLaVA and
  GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
  planning in general and complex environments.
paper_ai_keywords: embodied agents commonsense large language models (LLM) semantic knowledge plan generation physical scene constraint executable plans multimodal dataset triplets indoor scenes instructions action plans prompts GPT-3.5 grounded plan tuning pre-trained LLMs open-vocabulary object detectors multi-view RGB images inference success rate LLaVA