NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
We introduce NATURAL PLAN, a realistic planning benchmark in natural language
containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar
Scheduling. We focus our evaluation on the planning capabilities of LLMs with
full information on the task, by providing outputs from tools such as Google
Flights, Google Maps, and Google Calendar as contexts to the models. This
eliminates the need for a tool-use environment for evaluating LLMs on Planning.
We observe that NATURAL PLAN is a challenging benchmark for state of the art
models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only
achieve 31.1% and 34.8% solve rate respectively. We find that model performance
drops drastically as the complexity of the problem increases: all models
perform below 5% when there are 10 cities, highlighting a significant gap in
planning in natural language for SoTA LLMs. We also conduct extensive ablation
studies on NATURAL PLAN to further shed light on the (in)effectiveness of
approaches such as self-correction, few-shot generalization, and in-context
planning with long-contexts on improving LLM planning.
66665cb03a64c705291b9f46
Huaixiu Steven Zheng
66665cb03a64c705291b9f47
Swaroop Mishra
admin_assigned
2024-06-10T10:22:24.320Z
61841feb7c380c66d40cad54
/avatars/7a79362a24b844860781fb0625527eae.svg
Swaroop Mishra
user
swaroop7
66665cb03a64c705291b9f48
Hugh Zhang
claimed_verified
2024-09-10T07:10:51.097Z
65d6a5f94c28026a003581b4
/avatars/c24c314eafbf6a885c586258ae87a47f.svg
Hugh Zhang
user
hugh-scale
66665cb03a64c705291b9f49
Xinyun Chen
admin_assigned
2024-06-10T10:22:49.183Z
64d0268001931c60161e026a
/avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
Xinyun Chen
user
xinyunchen
66665cb03a64c705291b9f4a
Minmin Chen
admin_assigned
2024-06-10T10:22:56.410Z
661592c224c1fd99488e7ede
/avatars/f0599b34e72c5a5a859558df04292ccb.svg
Minmin Chen
user
mchen24
66665cb03a64c705291b9f4b
Azade Nova
66665cb03a64c705291b9f4c
Le Hou
admin_assigned
2024-06-10T10:23:15.625Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
66665cb03a64c705291b9f4d
Heng-Tze Cheng
66665cb03a64c705291b9f4e
Quoc V. Le
66665cb03a64c705291b9f4f
Ed H. Chi
66665cb03a64c705291b9f50
Denny Zhou
admin_assigned
2024-06-10T10:23:51.534Z
65d4d62d493aeaf0bb295f1b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
Denny Zhou
user
dennyzhou
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
We introduce NATURAL PLAN, a realistic planning benchmark in natural language
containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar
Scheduling. We focus our evaluation on the planning capabilities of LLMs with
full information on the task, by providing outputs from tools such as Google
Flights, Google Maps, and Google Calendar as contexts to the models. This
eliminates the need for a tool-use environment for evaluating LLMs on Planning.
We observe that NATURAL PLAN is a challenging benchmark for state of the art
models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only
achieve 31.1% and 34.8% solve rate respectively. We find that model performance
drops drastically as the complexity of the problem increases: all models
perform below 5% when there are 10 cities, highlighting a significant gap in
planning in natural language for SoTA LLMs. We also conduct extensive ablation
studies on NATURAL PLAN to further shed light on the (in)effectiveness of
approaches such as self-correction, few-shot generalization, and in-context
planning with long-contexts on improving LLM planning.
NATURAL PLAN is a challenging benchmark for evaluating the planning capabilities of large language models using natural language tasks like Trip Planning, Meeting Planning, and Calendar Scheduling.
LATENT PLAN Trip Planning Meeting Planning Calendar Scheduling LLMs Google Flights Google Maps Google Calendar self-correction few-shot generalization in-context planning