title: TravelPlanner: A Benchmark for Real-World Planning with Language Agents
summary: Planning has been part of the core pursuit for artificial intelligence since
  its conception, but earlier AI agents mostly focused on constrained settings
  because many of the cognitive substrates necessary for human-level planning
  have been lacking. Recently, language agents powered by large language models
  (LLMs) have shown interesting capabilities such as tool use and reasoning. Are
  these language agents capable of planning in more complex settings that are out
  of the reach of prior AI agents? To advance this investigation, we propose
  TravelPlanner, a new planning benchmark that focuses on travel planning, a
  common real-world planning scenario. It provides a rich sandbox environment,
  various tools for accessing nearly four million data records, and 1,225
  meticulously curated planning intents and reference plans. Comprehensive
  evaluations show that the current language agents are not yet capable of
  handling such complex planning tasks-even GPT-4 only achieves a success rate of
  0.6%. Language agents struggle to stay on task, use the right tools to collect
  information, or keep track of multiple constraints. However, we note that the
  mere possibility for language agents to tackle such a complex problem is in
  itself non-trivial progress. TravelPlanner provides a challenging yet
  meaningful testbed for future language agents.
paper_authors._id: 65c047c1ffc0ba672fb05e38
paper_authors.name: Jian Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:49:09.328Z
paper_authors._id: 62d65139667051e0a29bffe7
paper_authors.avatarUrl: /avatars/0252aa2bcd4cf1c8e4b87e5f164b6da5.svg
paper_authors.fullname: Jian Xie
paper_authors.type: user
paper_authors.user: hsaest
paper_authors._id: 65c047c1ffc0ba672fb05e39
paper_authors.name: Kai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:08:08.315Z
paper_authors._id: 63e0a50242591dda0b9dca5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: drogozhang
paper_authors._id: 65c047c1ffc0ba672fb05e3a
paper_authors.name: Jiangjie Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:08:15.379Z
paper_authors._id: 606ed1884ffe81d1e03e81e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1639375346654-606ed1884ffe81d1e03e81e5.png
paper_authors.fullname: Jiangjie Chen
paper_authors.type: user
paper_authors.user: jiangjiechen
paper_authors._id: 65c047c1ffc0ba672fb05e3b
paper_authors.name: Tinghui Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:08:21.547Z
paper_authors._id: 643f9e2288d9d4488fd81c52
paper_authors.avatarUrl: /avatars/e589c9cbd47022883cf33d7555bee89c.svg
paper_authors.fullname: Tinghui Zhu
paper_authors.type: user
paper_authors.user: DarthZhu
paper_authors._id: 65c047c1ffc0ba672fb05e3c
paper_authors.name: Renze Lou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:08:59.619Z
paper_authors._id: 6434a6e8ea46c009904c617e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434a6e8ea46c009904c617e/7xrjdbIgj40FtB1UeVrld.jpeg
paper_authors.fullname: RENZE LOU
paper_authors.type: user
paper_authors.user: Reza8848
paper_authors._id: 65c047c1ffc0ba672fb05e3d
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:09:06.327Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 65c047c1ffc0ba672fb05e3e
paper_authors.name: Yanghua Xiao
paper_authors._id: 65c047c1ffc0ba672fb05e3f
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:22:18.094Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_title: TravelPlanner: A Benchmark for Real-World Planning with Language Agents
paper_summary: Planning has been part of the core pursuit for artificial intelligence since
  its conception, but earlier AI agents mostly focused on constrained settings
  because many of the cognitive substrates necessary for human-level planning
  have been lacking. Recently, language agents powered by large language models
  (LLMs) have shown interesting capabilities such as tool use and reasoning. Are
  these language agents capable of planning in more complex settings that are out
  of the reach of prior AI agents? To advance this investigation, we propose
  TravelPlanner, a new planning benchmark that focuses on travel planning, a
  common real-world planning scenario. It provides a rich sandbox environment,
  various tools for accessing nearly four million data records, and 1,225
  meticulously curated planning intents and reference plans. Comprehensive
  evaluations show that the current language agents are not yet capable of
  handling such complex planning tasks-even GPT-4 only achieves a success rate of
  0.6%. Language agents struggle to stay on task, use the right tools to collect
  information, or keep track of multiple constraints. However, we note that the
  mere possibility for language agents to tackle such a complex problem is in
  itself non-trivial progress. TravelPlanner provides a challenging yet
  meaningful testbed for future language agents.
paper_ai_summary: TravelPlanner is a new benchmark for evaluating the planning capabilities of language agents in complex, real-world travel planning scenarios, demonstrating that current models still struggle with information gathering and constraint management.
paper_ai_keywords: large language models tool use reasoning TravelPlanner travel planning benchmark data records planning intents reference plans GPT-4 constraint management