Embodied Task Planning with Large Language Models
Equipping embodied agents with commonsense is important for robots to
successfully complete complex human instructions in general environments.
Recent large language models (LLM) can embed rich semantic knowledge for agents
in plan generation of complex tasks, while they lack the information about the
realistic world and usually yield infeasible action sequences. In this paper,
we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
with physical scene constraint, where the agent generates executable plans
according to the existed objects in the scene by aligning LLMs with the visual
perception models. Specifically, we first construct a multimodal dataset
containing triplets of indoor scenes, instructions and action plans, where we
provide the designed prompts and the list of existing objects in the scene for
GPT-3.5 to generate a large number of instructions and corresponding planned
actions. The generated data is leveraged for grounded plan tuning of
pre-trained LLMs. During inference, we discover the objects in the scene by
extending open-vocabulary object detectors to multi-view RGB images collected
in different achievable locations. Experimental results show that the generated
plan from our TaPA framework can achieve higher success rate than LLaVA and
GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
planning in general and complex environments.
64a62489f9a6bc9db6b17d0a
Zhenyu Wu
64a62489f9a6bc9db6b17d0b
Ziwei Wang
64a62489f9a6bc9db6b17d0c
Xiuwei Xu
claimed_verified
2023-07-10T09:24:59.569Z
648ac65fd044b25978015634
/avatars/2278a66fdc953220e9f8fc0ccce3ff00.svg
Xiuwei Xu
user
xuxw98
64a62489f9a6bc9db6b17d0d
Jiwen Lu
64a62489f9a6bc9db6b17d0e
Haibin Yan
Embodied Task Planning with Large Language Models
Equipping embodied agents with commonsense is important for robots to
successfully complete complex human instructions in general environments.
Recent large language models (LLM) can embed rich semantic knowledge for agents
in plan generation of complex tasks, while they lack the information about the
realistic world and usually yield infeasible action sequences. In this paper,
we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
with physical scene constraint, where the agent generates executable plans
according to the existed objects in the scene by aligning LLMs with the visual
perception models. Specifically, we first construct a multimodal dataset
containing triplets of indoor scenes, instructions and action plans, where we
provide the designed prompts and the list of existing objects in the scene for
GPT-3.5 to generate a large number of instructions and corresponding planned
actions. The generated data is leveraged for grounded plan tuning of
pre-trained LLMs. During inference, we discover the objects in the scene by
extending open-vocabulary object detectors to multi-view RGB images collected
in different achievable locations. Experimental results show that the generated
plan from our TaPA framework can achieve higher success rate than LLaVA and
GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
planning in general and complex environments.
TaPA integrates LLMs with visual perception to generate feasible and effective plans for embodied agents in complex environments, outperforming other models like LLaVA and GPT-3.5.
TAsk Planing Agent (TaPA) grounded planning physical scene constraint multimodal dataset triplets indoor scenes instructions action plans prompts GPT-3.5 grounded plan tuning pre-trained LLMs open-vocabulary object detectors multi-view RGB images experimental results feasibility embodied task planning