title: Which Agent Causes Task Failures and When? On Automated Failure
    Attribution of LLM Multi-Agent Systems
summary: Failure attribution in LLM multi-agent systems-identifying the agent and step
  responsible for task failures-provides crucial clues for systems debugging but
  remains underexplored and labor-intensive. In this paper, we propose and
  formulate a new research area: automated failure attribution for LLM
  multi-agent systems. To support this initiative, we introduce the Who&When
  dataset, comprising extensive failure logs from 127 LLM multi-agent systems
  with fine-grained annotations linking failures to specific agents and decisive
  error steps. Using the Who&When, we develop and evaluate three automated
  failure attribution methods, summarizing their corresponding pros and cons. The
  best method achieves 53.5% accuracy in identifying failure-responsible agents
  but only 14.2% in pinpointing failure steps, with some methods performing below
  random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to
  achieve practical usability. These results highlight the task's complexity and
  the need for further research in this area. Code and dataset are available at
  https://github.com/mingyin1/Agents_Failure_Attribution
paper_authors._id: 6814f3cf3d5bb41b95d414d1
paper_authors.name: Shaokun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:13:12.318Z
paper_authors._id: 65529f8b777d96ca1f87a59a
paper_authors.avatarUrl: /avatars/a78c783068d32de7c2b54401a1164cc2.svg
paper_authors.fullname: shaokun zhang
paper_authors.type: user
paper_authors.user: Shaokunkun
paper_authors._id: 6814f3cf3d5bb41b95d414d2
paper_authors.name: Ming Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-07T08:09:50.942Z
paper_authors._id: 66d4f80bbb76fb26ee6298cf
paper_authors.avatarUrl: /avatars/b341445ecc7f9b1c63b4bcba94d83bd0.svg
paper_authors.fullname: Ming Yin
paper_authors.type: user
paper_authors.user: Kevin355
paper_authors._id: 6814f3cf3d5bb41b95d414d3
paper_authors.name: Jieyu Zhang
paper_authors._id: 6814f3cf3d5bb41b95d414d4
paper_authors.name: Jiale Liu
paper_authors._id: 6814f3cf3d5bb41b95d414d5
paper_authors.name: Zhiguang Han
paper_authors._id: 6814f3cf3d5bb41b95d414d6
paper_authors.name: Jingyang Zhang
paper_authors._id: 6814f3cf3d5bb41b95d414d7
paper_authors.name: Beibin Li
paper_authors._id: 6814f3cf3d5bb41b95d414d8
paper_authors.name: Chi Wang
paper_authors._id: 6814f3cf3d5bb41b95d414d9
paper_authors.name: Huazheng Wang
paper_authors._id: 6814f3cf3d5bb41b95d414da
paper_authors.name: Yiran Chen
paper_authors._id: 6814f3cf3d5bb41b95d414db
paper_authors.name: Qingyun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T07:12:58.404Z
paper_authors._id: 647fa485d0cd8be13e662973
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647fa485d0cd8be13e662973/_QLgqLWhWir8_bkDCdTx6.png
paper_authors.fullname: Qingyun Wu
paper_authors.type: user
paper_authors.user: qingyun-wu
paper_title: Which Agent Causes Task Failures and When? On Automated Failure
    Attribution of LLM Multi-Agent Systems
paper_summary: Failure attribution in LLM multi-agent systems-identifying the agent and step
  responsible for task failures-provides crucial clues for systems debugging but
  remains underexplored and labor-intensive. In this paper, we propose and
  formulate a new research area: automated failure attribution for LLM
  multi-agent systems. To support this initiative, we introduce the Who&When
  dataset, comprising extensive failure logs from 127 LLM multi-agent systems
  with fine-grained annotations linking failures to specific agents and decisive
  error steps. Using the Who&When, we develop and evaluate three automated
  failure attribution methods, summarizing their corresponding pros and cons. The
  best method achieves 53.5% accuracy in identifying failure-responsible agents
  but only 14.2% in pinpointing failure steps, with some methods performing below
  random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to
  achieve practical usability. These results highlight the task's complexity and
  the need for further research in this area. Code and dataset are available at
  https://github.com/mingyin1/Agents_Failure_Attribution
paper_ai_summary: Automated failure attribution in LLM multi-agent systems is proposed using the Who&When dataset, with the best method achieving moderate agent accuracy but poor step accuracy.
paper_ai_keywords: LLM multi-agent systems failure attribution Who&When dataset automated failure attribution reasoning models OpenAI DeepSeek