title: Benchmarking LLMs' Swarm intelligence
summary: Large Language Models (LLMs) show potential for complex reasoning, yet their
  capacity for emergent coordination in Multi-Agent Systems (MAS) when operating
  under strict constraints-such as limited local perception and communication,
  characteristic of natural swarms-remains largely unexplored, particularly
  concerning the nuances of swarm intelligence. Existing benchmarks often do not
  fully capture the unique challenges of decentralized coordination that arise
  when agents operate with incomplete spatio-temporal information. To bridge this
  gap, we introduce SwarmBench, a novel benchmark designed to systematically
  evaluate the swarm intelligence capabilities of LLMs acting as decentralized
  agents. SwarmBench features five foundational MAS coordination tasks within a
  configurable 2D grid environment, forcing agents to rely primarily on local
  sensory input (k x k view) and local communication. We propose metrics for
  coordination effectiveness and analyze emergent group dynamics. Evaluating
  several leading LLMs in a zero-shot setting, we find significant performance
  variations across tasks, highlighting the difficulties posed by local
  information constraints. While some coordination emerges, results indicate
  limitations in robust planning and strategy formation under uncertainty in
  these decentralized scenarios. Assessing LLMs under swarm-like conditions is
  crucial for realizing their potential in future decentralized systems. We
  release SwarmBench as an open, extensible toolkit-built upon a customizable and
  scalable physical system with defined mechanical properties. It provides
  environments, prompts, evaluation scripts, and the comprehensive experimental
  datasets generated, aiming to foster reproducible research into LLM-based MAS
  coordination and the theoretical underpinnings of Embodied MAS. Our code
  repository is available at https://github.com/x66ccff/swarmbench.
paper_authors._id: 681c189c791c72783efe5a94
paper_authors.name: Kai Ruan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T08:58:58.134Z
paper_authors._id: 6205fefd3f1dc8a642d70b10
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673948194938-6205fefd3f1dc8a642d70b10.jpeg
paper_authors.fullname: Kai Ruan
paper_authors.type: user
paper_authors.user: 6cf
paper_authors._id: 681c189c791c72783efe5a95
paper_authors.name: Mowen Huang
paper_authors._id: 681c189c791c72783efe5a96
paper_authors.name: Ji-Rong Wen
paper_authors._id: 681c189c791c72783efe5a97
paper_authors.name: Hao Sun
paper_title: Benchmarking LLMs' Swarm intelligence
paper_summary: Large Language Models (LLMs) show potential for complex reasoning, yet their
  capacity for emergent coordination in Multi-Agent Systems (MAS) when operating
  under strict constraints-such as limited local perception and communication,
  characteristic of natural swarms-remains largely unexplored, particularly
  concerning the nuances of swarm intelligence. Existing benchmarks often do not
  fully capture the unique challenges of decentralized coordination that arise
  when agents operate with incomplete spatio-temporal information. To bridge this
  gap, we introduce SwarmBench, a novel benchmark designed to systematically
  evaluate the swarm intelligence capabilities of LLMs acting as decentralized
  agents. SwarmBench features five foundational MAS coordination tasks within a
  configurable 2D grid environment, forcing agents to rely primarily on local
  sensory input (k x k view) and local communication. We propose metrics for
  coordination effectiveness and analyze emergent group dynamics. Evaluating
  several leading LLMs in a zero-shot setting, we find significant performance
  variations across tasks, highlighting the difficulties posed by local
  information constraints. While some coordination emerges, results indicate
  limitations in robust planning and strategy formation under uncertainty in
  these decentralized scenarios. Assessing LLMs under swarm-like conditions is
  crucial for realizing their potential in future decentralized systems. We
  release SwarmBench as an open, extensible toolkit-built upon a customizable and
  scalable physical system with defined mechanical properties. It provides
  environments, prompts, evaluation scripts, and the comprehensive experimental
  datasets generated, aiming to foster reproducible research into LLM-based MAS
  coordination and the theoretical underpinnings of Embodied MAS. Our code
  repository is available at https://github.com/x66ccff/swarmbench.
paper_ai_summary: New benchmark evaluates LLMs in decentralized coordination tasks under limited information, highlighting challenges and potential for future systems.
paper_ai_keywords: Large Language Models Multi-Agent Systems SwarmBench decentralized coordination local perception local communication swarm intelligence coordination tasks 2D grid environment k x k view coordination effectiveness emergent group dynamics zero-shot evaluation robust planning strategy formation uncertainty Embodied MAS