title: Revealing the Barriers of Language Agents in Planning
summary: Autonomous planning has been an ongoing pursuit since the inception of
  artificial intelligence. Based on curated problem solvers, early planning
  agents could deliver precise solutions for specific tasks but lacked
  generalization. The emergence of large language models (LLMs) and their
  powerful reasoning capabilities has reignited interest in autonomous planning
  by automatically generating reasonable solutions for given tasks. However,
  prior research and our experiments show that current language agents still lack
  human-level planning abilities. Even the state-of-the-art reasoning model,
  OpenAI o1, achieves only 15.6% on one of the complex real-world planning
  benchmarks. This highlights a critical question: What hinders language agents
  from achieving human-level planning? Although existing studies have highlighted
  weak performance in agent planning, the deeper underlying issues and the
  mechanisms and limitations of the strategies proposed to address them remain
  insufficiently understood. In this work, we apply the feature attribution study
  and identify two key factors that hinder agent planning: the limited role of
  constraints and the diminishing influence of questions. We also find that
  although current strategies help mitigate these challenges, they do not fully
  resolve them, indicating that agents still have a long way to go before
  reaching human-level intelligence.
paper_authors._id: 67106ab8636030421b1ad24b
paper_authors.name: Jian Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:25:59.002Z
paper_authors._id: 62d65139667051e0a29bffe7
paper_authors.avatarUrl: /avatars/0252aa2bcd4cf1c8e4b87e5f164b6da5.svg
paper_authors.fullname: Jian Xie
paper_authors.type: user
paper_authors.user: hsaest
paper_authors._id: 67106ab8636030421b1ad24c
paper_authors.name: Kexun Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-17T01:39:05.772Z
paper_authors._id: 6074ff65cb07b90d90c9e5ac
paper_authors.avatarUrl: /avatars/770822e287821f8c6b284388f4145b93.svg
paper_authors.fullname: Kexun Zhang
paper_authors.type: user
paper_authors.user: kexunz
paper_authors._id: 67106ab8636030421b1ad24d
paper_authors.name: Jiangjie Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:25:56.929Z
paper_authors._id: 606ed1884ffe81d1e03e81e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1639375346654-606ed1884ffe81d1e03e81e5.png
paper_authors.fullname: Jiangjie Chen
paper_authors.type: user
paper_authors.user: jiangjiechen
paper_authors._id: 67106ab8636030421b1ad24e
paper_authors.name: Siyu Yuan
paper_authors._id: 67106ab8636030421b1ad24f
paper_authors.name: Kai Zhang
paper_authors._id: 67106ab8636030421b1ad250
paper_authors.name: Yikai Zhang
paper_authors._id: 67106ab8636030421b1ad251
paper_authors.name: Lei Li
paper_authors._id: 67106ab8636030421b1ad252
paper_authors.name: Yanghua Xiao
paper_title: Revealing the Barriers of Language Agents in Planning
paper_summary: Autonomous planning has been an ongoing pursuit since the inception of
  artificial intelligence. Based on curated problem solvers, early planning
  agents could deliver precise solutions for specific tasks but lacked
  generalization. The emergence of large language models (LLMs) and their
  powerful reasoning capabilities has reignited interest in autonomous planning
  by automatically generating reasonable solutions for given tasks. However,
  prior research and our experiments show that current language agents still lack
  human-level planning abilities. Even the state-of-the-art reasoning model,
  OpenAI o1, achieves only 15.6% on one of the complex real-world planning
  benchmarks. This highlights a critical question: What hinders language agents
  from achieving human-level planning? Although existing studies have highlighted
  weak performance in agent planning, the deeper underlying issues and the
  mechanisms and limitations of the strategies proposed to address them remain
  insufficiently understood. In this work, we apply the feature attribution study
  and identify two key factors that hinder agent planning: the limited role of
  constraints and the diminishing influence of questions. We also find that
  although current strategies help mitigate these challenges, they do not fully
  resolve them, indicating that agents still have a long way to go before
  reaching human-level intelligence.