TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Planning has been part of the core pursuit for artificial intelligence since
its conception, but earlier AI agents mostly focused on constrained settings
because many of the cognitive substrates necessary for human-level planning
have been lacking. Recently, language agents powered by large language models
(LLMs) have shown interesting capabilities such as tool use and reasoning. Are
these language agents capable of planning in more complex settings that are out
of the reach of prior AI agents? To advance this investigation, we propose
TravelPlanner, a new planning benchmark that focuses on travel planning, a
common real-world planning scenario. It provides a rich sandbox environment,
various tools for accessing nearly four million data records, and 1,225
meticulously curated planning intents and reference plans. Comprehensive
evaluations show that the current language agents are not yet capable of
handling such complex planning tasks-even GPT-4 only achieves a success rate of
0.6%. Language agents struggle to stay on task, use the right tools to collect
information, or keep track of multiple constraints. However, we note that the
mere possibility for language agents to tackle such a complex problem is in
itself non-trivial progress. TravelPlanner provides a challenging yet
meaningful testbed for future language agents.
65c047c1ffc0ba672fb05e38
Jian Xie
claimed_verified
2024-02-05T10:49:09.328Z
62d65139667051e0a29bffe7
/avatars/0252aa2bcd4cf1c8e4b87e5f164b6da5.svg
Jian Xie
user
hsaest
65c047c1ffc0ba672fb05e39
Kai Zhang
admin_assigned
2024-02-05T12:08:08.315Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
65c047c1ffc0ba672fb05e3a
Jiangjie Chen
admin_assigned
2024-02-05T12:08:15.379Z
606ed1884ffe81d1e03e81e5
https://cdn-avatars.huggingface.co/v1/production/uploads/1639375346654-606ed1884ffe81d1e03e81e5.png
Jiangjie Chen
user
jiangjiechen
65c047c1ffc0ba672fb05e3b
Tinghui Zhu
admin_assigned
2024-02-05T12:08:21.547Z
643f9e2288d9d4488fd81c52
/avatars/e589c9cbd47022883cf33d7555bee89c.svg
Tinghui Zhu
user
DarthZhu
65c047c1ffc0ba672fb05e3c
Renze Lou
admin_assigned
2024-02-05T12:08:59.619Z
6434a6e8ea46c009904c617e
https://cdn-avatars.huggingface.co/v1/production/uploads/6434a6e8ea46c009904c617e/7xrjdbIgj40FtB1UeVrld.jpeg
RENZE LOU
user
Reza8848
65c047c1ffc0ba672fb05e3d
Yuandong Tian
admin_assigned
2024-02-05T12:09:06.327Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
65c047c1ffc0ba672fb05e3e
Yanghua Xiao
65c047c1ffc0ba672fb05e3f
Yu Su
claimed_verified
2024-10-14T19:22:18.094Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Planning has been part of the core pursuit for artificial intelligence since
its conception, but earlier AI agents mostly focused on constrained settings
because many of the cognitive substrates necessary for human-level planning
have been lacking. Recently, language agents powered by large language models
(LLMs) have shown interesting capabilities such as tool use and reasoning. Are
these language agents capable of planning in more complex settings that are out
of the reach of prior AI agents? To advance this investigation, we propose
TravelPlanner, a new planning benchmark that focuses on travel planning, a
common real-world planning scenario. It provides a rich sandbox environment,
various tools for accessing nearly four million data records, and 1,225
meticulously curated planning intents and reference plans. Comprehensive
evaluations show that the current language agents are not yet capable of
handling such complex planning tasks-even GPT-4 only achieves a success rate of
0.6%. Language agents struggle to stay on task, use the right tools to collect
information, or keep track of multiple constraints. However, we note that the
mere possibility for language agents to tackle such a complex problem is in
itself non-trivial progress. TravelPlanner provides a challenging yet
meaningful testbed for future language agents.
TravelPlanner is a new benchmark for evaluating the planning capabilities of language agents in complex, real-world travel planning scenarios, demonstrating that current models still struggle with information gathering and constraint management.
large language models tool use reasoning TravelPlanner travel planning benchmark data records planning intents reference plans GPT-4 constraint management