Revealing the Barriers of Language Agents in Planning
Autonomous planning has been an ongoing pursuit since the inception of
artificial intelligence. Based on curated problem solvers, early planning
agents could deliver precise solutions for specific tasks but lacked
generalization. The emergence of large language models (LLMs) and their
powerful reasoning capabilities has reignited interest in autonomous planning
by automatically generating reasonable solutions for given tasks. However,
prior research and our experiments show that current language agents still lack
human-level planning abilities. Even the state-of-the-art reasoning model,
OpenAI o1, achieves only 15.6% on one of the complex real-world planning
benchmarks. This highlights a critical question: What hinders language agents
from achieving human-level planning? Although existing studies have highlighted
weak performance in agent planning, the deeper underlying issues and the
mechanisms and limitations of the strategies proposed to address them remain
insufficiently understood. In this work, we apply the feature attribution study
and identify two key factors that hinder agent planning: the limited role of
constraints and the diminishing influence of questions. We also find that
although current strategies help mitigate these challenges, they do not fully
resolve them, indicating that agents still have a long way to go before
reaching human-level intelligence.
67106ab8636030421b1ad24b
Jian Xie
claimed_verified
2024-10-17T11:25:59.002Z
62d65139667051e0a29bffe7
/avatars/0252aa2bcd4cf1c8e4b87e5f164b6da5.svg
Jian Xie
user
hsaest
67106ab8636030421b1ad24c
Kexun Zhang
extracted_pending
2024-10-17T01:39:05.772Z
6074ff65cb07b90d90c9e5ac
/avatars/770822e287821f8c6b284388f4145b93.svg
Kexun Zhang
user
kexunz
67106ab8636030421b1ad24d
Jiangjie Chen
claimed_verified
2024-10-17T11:25:56.929Z
606ed1884ffe81d1e03e81e5
https://cdn-avatars.huggingface.co/v1/production/uploads/1639375346654-606ed1884ffe81d1e03e81e5.png
Jiangjie Chen
user
jiangjiechen
67106ab8636030421b1ad24e
Siyu Yuan
67106ab8636030421b1ad24f
Kai Zhang
67106ab8636030421b1ad250
Yikai Zhang
67106ab8636030421b1ad251
Lei Li
67106ab8636030421b1ad252
Yanghua Xiao
Revealing the Barriers of Language Agents in Planning
Autonomous planning has been an ongoing pursuit since the inception of
artificial intelligence. Based on curated problem solvers, early planning
agents could deliver precise solutions for specific tasks but lacked
generalization. The emergence of large language models (LLMs) and their
powerful reasoning capabilities has reignited interest in autonomous planning
by automatically generating reasonable solutions for given tasks. However,
prior research and our experiments show that current language agents still lack
human-level planning abilities. Even the state-of-the-art reasoning model,
OpenAI o1, achieves only 15.6% on one of the complex real-world planning
benchmarks. This highlights a critical question: What hinders language agents
from achieving human-level planning? Although existing studies have highlighted
weak performance in agent planning, the deeper underlying issues and the
mechanisms and limitations of the strategies proposed to address them remain
insufficiently understood. In this work, we apply the feature attribution study
and identify two key factors that hinder agent planning: the limited role of
constraints and the diminishing influence of questions. We also find that
although current strategies help mitigate these challenges, they do not fully
resolve them, indicating that agents still have a long way to go before
reaching human-level intelligence.
Research identifies constraints and question influence as critical factors hindering the planning capabilities of current language agents.
large language models reasoning capabilities autonomous planning feature attribution constraints questions human-level intelligence