title: Why Do Multi-Agent LLM Systems Fail?
summary: Despite growing enthusiasm for Multi-Agent Systems (MAS), where multiple LLM
  agents collaborate to accomplish tasks, their performance gains across popular
  benchmarks remain minimal compared to single-agent frameworks. This gap
  highlights the need to analyze the challenges hindering MAS effectiveness.
    In this paper, we present the first comprehensive study of MAS challenges. We
  analyze five popular MAS frameworks across over 150 tasks, involving six expert
  human annotators. We identify 14 unique failure modes and propose a
  comprehensive taxonomy applicable to various MAS frameworks. This taxonomy
  emerges iteratively from agreements among three expert annotators per study,
  achieving a Cohen's Kappa score of 0.88. These fine-grained failure modes are
  organized into 3 categories, (i) specification and system design failures, (ii)
  inter-agent misalignment, and (iii) task verification and termination. To
  support scalable evaluation, we integrate MASFT with LLM-as-a-Judge. We also
  explore if identified failures could be easily prevented by proposing two
  interventions: improved specification of agent roles and enhanced orchestration
  strategies. Our findings reveal that identified failures require more complex
  solutions, highlighting a clear roadmap for future research. We open-source our
  dataset and LLM annotator.
paper_authors._id: 67dc4391f618f3c7ba6a3b6d
paper_authors.name: Mert Cemri
paper_authors._id: 67dc4391f618f3c7ba6a3b6e
paper_authors.name: Melissa Z. Pan
paper_authors._id: 67dc4391f618f3c7ba6a3b6f
paper_authors.name: Shuyi Yang
paper_authors._id: 67dc4391f618f3c7ba6a3b70
paper_authors.name: Lakshya A. Agrawal
paper_authors._id: 67dc4391f618f3c7ba6a3b71
paper_authors.name: Bhavya Chopra
paper_authors._id: 67dc4391f618f3c7ba6a3b72
paper_authors.name: Rishabh Tiwari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:54:39.712Z
paper_authors._id: 66db38c38d2688295f731283
paper_authors.avatarUrl: /avatars/a1f832d354a1f5d5c11593bf276b47a6.svg
paper_authors.fullname: Rishabh Tiwari
paper_authors.type: user
paper_authors.user: rishabh2k1
paper_authors._id: 67dc4391f618f3c7ba6a3b73
paper_authors.name: Kurt Keutzer
paper_authors._id: 67dc4391f618f3c7ba6a3b74
paper_authors.name: Aditya Parameswaran
paper_authors._id: 67dc4391f618f3c7ba6a3b75
paper_authors.name: Dan Klein
paper_authors._id: 67dc4391f618f3c7ba6a3b76
paper_authors.name: Kannan Ramchandran
paper_authors._id: 67dc4391f618f3c7ba6a3b77
paper_authors.name: Matei Zaharia
paper_authors._id: 67dc4391f618f3c7ba6a3b78
paper_authors.name: Joseph E. Gonzalez
paper_authors._id: 67dc4391f618f3c7ba6a3b79
paper_authors.name: Ion Stoica
paper_title: Why Do Multi-Agent LLM Systems Fail?
paper_summary: Despite growing enthusiasm for Multi-Agent Systems (MAS), where multiple LLM
  agents collaborate to accomplish tasks, their performance gains across popular
  benchmarks remain minimal compared to single-agent frameworks. This gap
  highlights the need to analyze the challenges hindering MAS effectiveness.
    In this paper, we present the first comprehensive study of MAS challenges. We
  analyze five popular MAS frameworks across over 150 tasks, involving six expert
  human annotators. We identify 14 unique failure modes and propose a
  comprehensive taxonomy applicable to various MAS frameworks. This taxonomy
  emerges iteratively from agreements among three expert annotators per study,
  achieving a Cohen's Kappa score of 0.88. These fine-grained failure modes are
  organized into 3 categories, (i) specification and system design failures, (ii)
  inter-agent misalignment, and (iii) task verification and termination. To
  support scalable evaluation, we integrate MASFT with LLM-as-a-Judge. We also
  explore if identified failures could be easily prevented by proposing two
  interventions: improved specification of agent roles and enhanced orchestration
  strategies. Our findings reveal that identified failures require more complex
  solutions, highlighting a clear roadmap for future research. We open-source our
  dataset and LLM annotator.
paper_ai_summary: A comprehensive study identifies 14 unique failure modes in Multi-Agent Systems, categorizing them into specification issues, inter-agent conflicts, and task verification problems, and proposes interventions for preventing these failures.
paper_ai_keywords: Multi-Agent Systems (MAS) LLM specification and system design failures inter-agent misalignment task verification and termination Cohen's Kappa LLM-as-a-Judge MASFT agent roles orchestration strategies