title: MultiAgentBench: Evaluating the Collaboration and Competition of LLM
    agents
summary: Large Language Models (LLMs) have shown remarkable capabilities as autonomous
  agents, yet existing benchmarks either focus on single-agent tasks or are
  confined to narrow domains, failing to capture the dynamics of multi-agent
  coordination and competition. In this paper, we introduce MultiAgentBench, a
  comprehensive benchmark designed to evaluate LLM-based multi-agent systems
  across diverse, interactive scenarios. Our framework measures not only task
  completion but also the quality of collaboration and competition using novel,
  milestone-based key performance indicators. Moreover, we evaluate various
  coordination protocols (including star, chain, tree, and graph topologies) and
  innovative strategies such as group discussion and cognitive planning. Notably,
  gpt-4o-mini reaches the average highest task score, graph structure performs
  the best among coordination protocols in the research scenario, and cognitive
  planning improves milestone achievement rates by 3%. Code and datasets are
  public available at https://github.com/MultiagentBench/MARBLE.
paper_authors._id: 67c7ba7f19b236e0564d1172
paper_authors.name: Kunlun Zhu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-05T02:44:18.739Z
paper_authors._id: 66554507e6ea63012f35824c
paper_authors.avatarUrl: /avatars/b82de75bd60890e7bb524fc3754b131c.svg
paper_authors.fullname: Kunlun_Zhu
paper_authors.type: user
paper_authors.user: Leozkl
paper_authors._id: 67c7ba7f19b236e0564d1173
paper_authors.name: Hongyi Du
paper_authors._id: 67c7ba7f19b236e0564d1174
paper_authors.name: Zhaochen Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:38:26.340Z
paper_authors._id: 672516b24c0a2bf36e52ec9d
paper_authors.avatarUrl: /avatars/ce20ec30586dd40d263fd0b88eaf002d.svg
paper_authors.fullname: Zhaochen Hong
paper_authors.type: user
paper_authors.user: zhaochenhong
paper_authors._id: 67c7ba7f19b236e0564d1175
paper_authors.name: Xiaocheng Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:38:17.378Z
paper_authors._id: 64f98e9c31b3625c3844b73d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f98e9c31b3625c3844b73d/if9kvZUkXTuo9UuzJY236.jpeg
paper_authors.fullname: Xiaocheng Yang
paper_authors.type: user
paper_authors.user: Thomas-X-Yang
paper_authors._id: 67c7ba7f19b236e0564d1176
paper_authors.name: Shuyi Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:38:47.235Z
paper_authors._id: 65ca8d92d5822659ecc6c0d4
paper_authors.avatarUrl: /avatars/6e9ab0470d68248a267eec892fce89c3.svg
paper_authors.fullname: Shuyi
paper_authors.type: user
paper_authors.user: ShuyiGuo
paper_authors._id: 67c7ba7f19b236e0564d1177
paper_authors.name: Zhe Wang
paper_authors._id: 67c7ba7f19b236e0564d1178
paper_authors.name: Zhenhailong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:34:51.267Z
paper_authors._id: 628d7265db4cd1d1717c884f
paper_authors.avatarUrl: /avatars/dff2a3dd10d84b4a73fa486402de7219.svg
paper_authors.fullname: Zhenhailong Wang
paper_authors.type: user
paper_authors.user: mikewang
paper_authors._id: 67c7ba7f19b236e0564d1179
paper_authors.name: Cheng Qian
paper_authors._id: 67c7ba7f19b236e0564d117a
paper_authors.name: Xiangru Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:35:40.308Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 67c7ba7f19b236e0564d117b
paper_authors.name: Heng Ji
paper_authors._id: 67c7ba7f19b236e0564d117c
paper_authors.name: Jiaxuan You
paper_title: MultiAgentBench: Evaluating the Collaboration and Competition of LLM
    agents
paper_summary: Large Language Models (LLMs) have shown remarkable capabilities as autonomous
  agents, yet existing benchmarks either focus on single-agent tasks or are
  confined to narrow domains, failing to capture the dynamics of multi-agent
  coordination and competition. In this paper, we introduce MultiAgentBench, a
  comprehensive benchmark designed to evaluate LLM-based multi-agent systems
  across diverse, interactive scenarios. Our framework measures not only task
  completion but also the quality of collaboration and competition using novel,
  milestone-based key performance indicators. Moreover, we evaluate various
  coordination protocols (including star, chain, tree, and graph topologies) and
  innovative strategies such as group discussion and cognitive planning. Notably,
  gpt-4o-mini reaches the average highest task score, graph structure performs
  the best among coordination protocols in the research scenario, and cognitive
  planning improves milestone achievement rates by 3%. Code and datasets are
  public available at https://github.com/MultiagentBench/MARBLE.
paper_ai_keywords: MultiAgentBench multi-agent systems milestone-based key performance indicators coordination protocols star topology chain topology tree topology graph topology group discussion cognitive planning task score milestone achievement rates