title: NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
summary: We introduce NATURAL PLAN, a realistic planning benchmark in natural language
  containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar
  Scheduling. We focus our evaluation on the planning capabilities of LLMs with
  full information on the task, by providing outputs from tools such as Google
  Flights, Google Maps, and Google Calendar as contexts to the models. This
  eliminates the need for a tool-use environment for evaluating LLMs on Planning.
  We observe that NATURAL PLAN is a challenging benchmark for state of the art
  models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only
  achieve 31.1% and 34.8% solve rate respectively. We find that model performance
  drops drastically as the complexity of the problem increases: all models
  perform below 5% when there are 10 cities, highlighting a significant gap in
  planning in natural language for SoTA LLMs. We also conduct extensive ablation
  studies on NATURAL PLAN to further shed light on the (in)effectiveness of
  approaches such as self-correction, few-shot generalization, and in-context
  planning with long-contexts on improving LLM planning.
paper_authors._id: 66665cb03a64c705291b9f46
paper_authors.name: Huaixiu Steven Zheng
paper_authors._id: 66665cb03a64c705291b9f47
paper_authors.name: Swaroop Mishra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:22:24.320Z
paper_authors._id: 61841feb7c380c66d40cad54
paper_authors.avatarUrl: /avatars/7a79362a24b844860781fb0625527eae.svg
paper_authors.fullname: Swaroop Mishra
paper_authors.type: user
paper_authors.user: swaroop7
paper_authors._id: 66665cb03a64c705291b9f48
paper_authors.name: Hugh Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T07:10:51.097Z
paper_authors._id: 65d6a5f94c28026a003581b4
paper_authors.avatarUrl: /avatars/c24c314eafbf6a885c586258ae87a47f.svg
paper_authors.fullname: Hugh Zhang
paper_authors.type: user
paper_authors.user: hugh-scale
paper_authors._id: 66665cb03a64c705291b9f49
paper_authors.name: Xinyun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:22:49.183Z
paper_authors._id: 64d0268001931c60161e026a
paper_authors.avatarUrl: /avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
paper_authors.fullname: Xinyun Chen
paper_authors.type: user
paper_authors.user: xinyunchen
paper_authors._id: 66665cb03a64c705291b9f4a
paper_authors.name: Minmin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:22:56.410Z
paper_authors._id: 661592c224c1fd99488e7ede
paper_authors.avatarUrl: /avatars/f0599b34e72c5a5a859558df04292ccb.svg
paper_authors.fullname: Minmin Chen
paper_authors.type: user
paper_authors.user: mchen24
paper_authors._id: 66665cb03a64c705291b9f4b
paper_authors.name: Azade Nova
paper_authors._id: 66665cb03a64c705291b9f4c
paper_authors.name: Le Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:23:15.625Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 66665cb03a64c705291b9f4d
paper_authors.name: Heng-Tze Cheng
paper_authors._id: 66665cb03a64c705291b9f4e
paper_authors.name: Quoc V. Le
paper_authors._id: 66665cb03a64c705291b9f4f
paper_authors.name: Ed H. Chi
paper_authors._id: 66665cb03a64c705291b9f50
paper_authors.name: Denny Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:23:51.534Z
paper_authors._id: 65d4d62d493aeaf0bb295f1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
paper_authors.fullname: Denny Zhou
paper_authors.type: user
paper_authors.user: dennyzhou
paper_title: NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
paper_summary: We introduce NATURAL PLAN, a realistic planning benchmark in natural language
  containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar
  Scheduling. We focus our evaluation on the planning capabilities of LLMs with
  full information on the task, by providing outputs from tools such as Google
  Flights, Google Maps, and Google Calendar as contexts to the models. This
  eliminates the need for a tool-use environment for evaluating LLMs on Planning.
  We observe that NATURAL PLAN is a challenging benchmark for state of the art
  models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only
  achieve 31.1% and 34.8% solve rate respectively. We find that model performance
  drops drastically as the complexity of the problem increases: all models
  perform below 5% when there are 10 cities, highlighting a significant gap in
  planning in natural language for SoTA LLMs. We also conduct extensive ablation
  studies on NATURAL PLAN to further shed light on the (in)effectiveness of
  approaches such as self-correction, few-shot generalization, and in-context
  planning with long-contexts on improving LLM planning.
paper_ai_summary: NATURAL PLAN is a challenging benchmark for evaluating the planning capabilities of large language models using natural language tasks like Trip Planning, Meeting Planning, and Calendar Scheduling.
paper_ai_keywords: LATENT PLAN Trip Planning Meeting Planning Calendar Scheduling LLMs Google Flights Google Maps Google Calendar self-correction few-shot generalization in-context planning