Which Agent Causes Task Failures and When? On Automated Failure
  Attribution of LLM Multi-Agent Systems
Failure attribution in LLM multi-agent systems-identifying the agent and step
responsible for task failures-provides crucial clues for systems debugging but
remains underexplored and labor-intensive. In this paper, we propose and
formulate a new research area: automated failure attribution for LLM
multi-agent systems. To support this initiative, we introduce the Who&When
dataset, comprising extensive failure logs from 127 LLM multi-agent systems
with fine-grained annotations linking failures to specific agents and decisive
error steps. Using the Who&When, we develop and evaluate three automated
failure attribution methods, summarizing their corresponding pros and cons. The
best method achieves 53.5% accuracy in identifying failure-responsible agents
but only 14.2% in pinpointing failure steps, with some methods performing below
random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to
achieve practical usability. These results highlight the task's complexity and
the need for further research in this area. Code and dataset are available at
https://github.com/mingyin1/Agents_Failure_Attribution
6814f3cf3d5bb41b95d414d1
Shaokun Zhang
claimed_verified
2025-05-13T08:13:12.318Z
65529f8b777d96ca1f87a59a
/avatars/a78c783068d32de7c2b54401a1164cc2.svg
shaokun zhang
user
Shaokunkun
6814f3cf3d5bb41b95d414d2
Ming Yin
claimed_verified
2025-05-07T08:09:50.942Z
66d4f80bbb76fb26ee6298cf
/avatars/b341445ecc7f9b1c63b4bcba94d83bd0.svg
Ming Yin
user
Kevin355
6814f3cf3d5bb41b95d414d3
Jieyu Zhang
6814f3cf3d5bb41b95d414d4
Jiale Liu
6814f3cf3d5bb41b95d414d5
Zhiguang Han
6814f3cf3d5bb41b95d414d6
Jingyang Zhang
6814f3cf3d5bb41b95d414d7
Beibin Li
6814f3cf3d5bb41b95d414d8
Chi Wang
6814f3cf3d5bb41b95d414d9
Huazheng Wang
claimed_verified
2025-07-30T09:05:33.811Z
688969c4d0ac71994caa0df6
/avatars/f8f89e0ca810908cfe21c921d8373e28.svg
Huazheng Wang
user
huazhengwang
6814f3cf3d5bb41b95d414da
Yiran Chen
6814f3cf3d5bb41b95d414db
Qingyun Wu
claimed_verified
2025-05-16T07:12:58.404Z
647fa485d0cd8be13e662973
https://cdn-avatars.huggingface.co/v1/production/uploads/647fa485d0cd8be13e662973/_QLgqLWhWir8_bkDCdTx6.png
Qingyun Wu
user
qingyun-wu
Which Agent Causes Task Failures and When? On Automated Failure
  Attribution of LLM Multi-Agent Systems
Failure attribution in LLM multi-agent systems-identifying the agent and step
responsible for task failures-provides crucial clues for systems debugging but
remains underexplored and labor-intensive. In this paper, we propose and
formulate a new research area: automated failure attribution for LLM
multi-agent systems. To support this initiative, we introduce the Who&When
dataset, comprising extensive failure logs from 127 LLM multi-agent systems
with fine-grained annotations linking failures to specific agents and decisive
error steps. Using the Who&When, we develop and evaluate three automated
failure attribution methods, summarizing their corresponding pros and cons. The
best method achieves 53.5% accuracy in identifying failure-responsible agents
but only 14.2% in pinpointing failure steps, with some methods performing below
random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to
achieve practical usability. These results highlight the task's complexity and
the need for further research in this area. Code and dataset are available at
https://github.com/mingyin1/Agents_Failure_Attribution
Automated failure attribution in LLM multi-agent systems is proposed using the Who&When dataset, with the best method achieving moderate agent accuracy but poor step accuracy.
LLM multi-agent systems failure attribution Who&When dataset automated failure attribution reasoning models OpenAI DeepSeek