SayPlan: Grounding Large Language Models using 3D Scene Graphs for
  Scalable Task Planning
Large language models (LLMs) have demonstrated impressive results in
developing generalist planning agents for diverse tasks. However, grounding
these plans in expansive, multi-floor, and multi-room environments presents a
significant challenge for robotics. We introduce SayPlan, a scalable approach
to LLM-based, large-scale task planning for robotics using 3D scene graph
(3DSG) representations. To ensure the scalability of our approach, we: (1)
exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic
search for task-relevant subgraphs from a smaller, collapsed representation of
the full graph; (2) reduce the planning horizon for the LLM by integrating a
classical path planner and (3) introduce an iterative replanning pipeline that
refines the initial plan using feedback from a scene graph simulator,
correcting infeasible actions and avoiding planning failures. We evaluate our
approach on two large-scale environments spanning up to 3 floors, 36 rooms and
140 objects, and show that our approach is capable of grounding large-scale,
long-horizon task plans from abstract, and natural language instruction for a
mobile manipulator robot to execute.
64af6fe870277aca9044f00d
Krishan Rana
claimed_verified
2023-07-13T05:22:09.660Z
631eef495ba8c0263407f0f1
/avatars/bec4dc1d5a61607397272732e5512329.svg
Krishan Rana
user
krishanrana
64af6fe870277aca9044f00e
Jesse Haviland
64af6fe870277aca9044f00f
Sourav Garg
admin_assigned
2023-07-13T09:13:39.282Z
630877951dd1e3075d95fc73
/avatars/0ef8e0763732a368d5b73d05c54f8e14.svg
Sourav Garg
user
oravus
64af6fe870277aca9044f010
Jad Abou-Chakra
64af6fe870277aca9044f011
Ian Reid
64af6fe870277aca9044f012
Niko Suenderhauf
admin_assigned
2023-07-13T09:14:40.365Z
62cf95bde1d5ee194e69dc9e
/avatars/80112f9cb3fcc2824061f627ca02e5e4.svg
Niko Suenderhauf
user
nikosuenderhauf
SayPlan: Grounding Large Language Models using 3D Scene Graphs for
  Scalable Task Planning
Large language models (LLMs) have demonstrated impressive results in
developing generalist planning agents for diverse tasks. However, grounding
these plans in expansive, multi-floor, and multi-room environments presents a
significant challenge for robotics. We introduce SayPlan, a scalable approach
to LLM-based, large-scale task planning for robotics using 3D scene graph
(3DSG) representations. To ensure the scalability of our approach, we: (1)
exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic
search for task-relevant subgraphs from a smaller, collapsed representation of
the full graph; (2) reduce the planning horizon for the LLM by integrating a
classical path planner and (3) introduce an iterative replanning pipeline that
refines the initial plan using feedback from a scene graph simulator,
correcting infeasible actions and avoiding planning failures. We evaluate our
approach on two large-scale environments spanning up to 3 floors, 36 rooms and
140 objects, and show that our approach is capable of grounding large-scale,
long-horizon task plans from abstract, and natural language instruction for a
mobile manipulator robot to execute.
SayPlan combines LLMs with 3D scene graphs to enable scalable and iterative task planning for robots in complex, large-scale environments.
large language models LLMs generalist planning agents 3D scene graph 3DSG semantic search task-relevant subgraphs classical path planner iterative replanning pipeline scene graph simulator infeasible actions planning failures mobile manipulator robot