Why Do Multi-Agent LLM Systems Fail?
Despite growing enthusiasm for Multi-Agent Systems (MAS), where multiple LLM
agents collaborate to accomplish tasks, their performance gains across popular
benchmarks remain minimal compared to single-agent frameworks. This gap
highlights the need to analyze the challenges hindering MAS effectiveness.
  In this paper, we present the first comprehensive study of MAS challenges. We
analyze five popular MAS frameworks across over 150 tasks, involving six expert
human annotators. We identify 14 unique failure modes and propose a
comprehensive taxonomy applicable to various MAS frameworks. This taxonomy
emerges iteratively from agreements among three expert annotators per study,
achieving a Cohen's Kappa score of 0.88. These fine-grained failure modes are
organized into 3 categories, (i) specification and system design failures, (ii)
inter-agent misalignment, and (iii) task verification and termination. To
support scalable evaluation, we integrate MASFT with LLM-as-a-Judge. We also
explore if identified failures could be easily prevented by proposing two
interventions: improved specification of agent roles and enhanced orchestration
strategies. Our findings reveal that identified failures require more complex
solutions, highlighting a clear roadmap for future research. We open-source our
dataset and LLM annotator.
67dc4391f618f3c7ba6a3b6d
Mert Cemri
67dc4391f618f3c7ba6a3b6e
Melissa Z. Pan
extracted_pending
2025-07-17T17:05:52.366Z
66253597f5c285535c91cfaf
/avatars/4d7385851aff97c0798577e081196554.svg
Melissa Pan
user
melissapan
67dc4391f618f3c7ba6a3b6f
Shuyi Yang
67dc4391f618f3c7ba6a3b70
Lakshya A. Agrawal
claimed_verified
2025-07-29T07:12:02.011Z
63bf0cbb24e75d1efdfdf8bd
/avatars/3bb13e989db31820734cbe6b13be130a.svg
Lakshya A Agrawal
user
LakshyAAAgrawal
67dc4391f618f3c7ba6a3b71
Bhavya Chopra
67dc4391f618f3c7ba6a3b72
Rishabh Tiwari
claimed_verified
2025-03-24T08:54:39.712Z
66db38c38d2688295f731283
/avatars/a1f832d354a1f5d5c11593bf276b47a6.svg
Rishabh Tiwari
user
rishabh2k1
67dc4391f618f3c7ba6a3b73
Kurt Keutzer
67dc4391f618f3c7ba6a3b74
Aditya Parameswaran
67dc4391f618f3c7ba6a3b75
Dan Klein
67dc4391f618f3c7ba6a3b76
Kannan Ramchandran
67dc4391f618f3c7ba6a3b77
Matei Zaharia
67dc4391f618f3c7ba6a3b78
Joseph E. Gonzalez
67dc4391f618f3c7ba6a3b79
Ion Stoica
Why Do Multi-Agent LLM Systems Fail?
Despite growing enthusiasm for Multi-Agent Systems (MAS), where multiple LLM
agents collaborate to accomplish tasks, their performance gains across popular
benchmarks remain minimal compared to single-agent frameworks. This gap
highlights the need to analyze the challenges hindering MAS effectiveness.
  In this paper, we present the first comprehensive study of MAS challenges. We
analyze five popular MAS frameworks across over 150 tasks, involving six expert
human annotators. We identify 14 unique failure modes and propose a
comprehensive taxonomy applicable to various MAS frameworks. This taxonomy
emerges iteratively from agreements among three expert annotators per study,
achieving a Cohen's Kappa score of 0.88. These fine-grained failure modes are
organized into 3 categories, (i) specification and system design failures, (ii)
inter-agent misalignment, and (iii) task verification and termination. To
support scalable evaluation, we integrate MASFT with LLM-as-a-Judge. We also
explore if identified failures could be easily prevented by proposing two
interventions: improved specification of agent roles and enhanced orchestration
strategies. Our findings reveal that identified failures require more complex
solutions, highlighting a clear roadmap for future research. We open-source our
dataset and LLM annotator.
A comprehensive study identifies 14 unique failure modes in Multi-Agent Systems, categorizing them into specification issues, inter-agent conflicts, and task verification problems, and proposes interventions for preventing these failures.
Multi-Agent Systems (MAS) LLM specification and system design failures inter-agent misalignment task verification and termination Cohen's Kappa LLM-as-a-Judge MASFT agent roles orchestration strategies