title: SayPlan: Grounding Large Language Models using 3D Scene Graphs for
    Scalable Task Planning
summary: Large language models (LLMs) have demonstrated impressive results in
  developing generalist planning agents for diverse tasks. However, grounding
  these plans in expansive, multi-floor, and multi-room environments presents a
  significant challenge for robotics. We introduce SayPlan, a scalable approach
  to LLM-based, large-scale task planning for robotics using 3D scene graph
  (3DSG) representations. To ensure the scalability of our approach, we: (1)
  exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic
  search for task-relevant subgraphs from a smaller, collapsed representation of
  the full graph; (2) reduce the planning horizon for the LLM by integrating a
  classical path planner and (3) introduce an iterative replanning pipeline that
  refines the initial plan using feedback from a scene graph simulator,
  correcting infeasible actions and avoiding planning failures. We evaluate our
  approach on two large-scale environments spanning up to 3 floors, 36 rooms and
  140 objects, and show that our approach is capable of grounding large-scale,
  long-horizon task plans from abstract, and natural language instruction for a
  mobile manipulator robot to execute.
paper_authors._id: 64af6fe870277aca9044f00d
paper_authors.name: Krishan Rana
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-13T05:22:09.660Z
paper_authors._id: 631eef495ba8c0263407f0f1
paper_authors.avatarUrl: /avatars/bec4dc1d5a61607397272732e5512329.svg
paper_authors.fullname: Krishan Rana
paper_authors.type: user
paper_authors.user: krishanrana
paper_authors._id: 64af6fe870277aca9044f00e
paper_authors.name: Jesse Haviland
paper_authors._id: 64af6fe870277aca9044f00f
paper_authors.name: Sourav Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:13:39.282Z
paper_authors._id: 630877951dd1e3075d95fc73
paper_authors.avatarUrl: /avatars/0ef8e0763732a368d5b73d05c54f8e14.svg
paper_authors.fullname: Sourav Garg
paper_authors.type: user
paper_authors.user: oravus
paper_authors._id: 64af6fe870277aca9044f010
paper_authors.name: Jad Abou-Chakra
paper_authors._id: 64af6fe870277aca9044f011
paper_authors.name: Ian Reid
paper_authors._id: 64af6fe870277aca9044f012
paper_authors.name: Niko Suenderhauf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:14:40.365Z
paper_authors._id: 62cf95bde1d5ee194e69dc9e
paper_authors.avatarUrl: /avatars/80112f9cb3fcc2824061f627ca02e5e4.svg
paper_authors.fullname: Niko Suenderhauf
paper_authors.type: user
paper_authors.user: nikosuenderhauf
paper_title: SayPlan: Grounding Large Language Models using 3D Scene Graphs for
    Scalable Task Planning
paper_summary: Large language models (LLMs) have demonstrated impressive results in
  developing generalist planning agents for diverse tasks. However, grounding
  these plans in expansive, multi-floor, and multi-room environments presents a
  significant challenge for robotics. We introduce SayPlan, a scalable approach
  to LLM-based, large-scale task planning for robotics using 3D scene graph
  (3DSG) representations. To ensure the scalability of our approach, we: (1)
  exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic
  search for task-relevant subgraphs from a smaller, collapsed representation of
  the full graph; (2) reduce the planning horizon for the LLM by integrating a
  classical path planner and (3) introduce an iterative replanning pipeline that
  refines the initial plan using feedback from a scene graph simulator,
  correcting infeasible actions and avoiding planning failures. We evaluate our
  approach on two large-scale environments spanning up to 3 floors, 36 rooms and
  140 objects, and show that our approach is capable of grounding large-scale,
  long-horizon task plans from abstract, and natural language instruction for a
  mobile manipulator robot to execute.
paper_ai_summary: SayPlan combines LLMs with 3D scene graphs to enable scalable and iterative task planning for robots in complex, large-scale environments.
paper_ai_keywords: large language models LLMs generalist planning agents 3D scene graph 3DSG semantic search task-relevant subgraphs classical path planner iterative replanning pipeline scene graph simulator infeasible actions planning failures mobile manipulator robot