title: SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?
summary: Reasoning and strategic behavior in social interactions is a hallmark
  of intelligence. This form of reasoning is significantly more sophisticated
  than isolated planning or reasoning tasks in static settings (e.g., math
  problem solving). In this paper, we present Strategic Planning,
  Interaction, and Negotiation (SPIN-Bench), a new multi-domain
  evaluation designed to measure the intelligence of strategic planning
  and social reasoning. While many existing benchmarks focus on narrow
  planning or single-agent reasoning, SPIN-Bench combines classical PDDL tasks,
  competitive board games, cooperative card games, and multi-agent negotiation
  scenarios in one unified framework. The framework includes both a benchmark as
  well as an arena to simulate and evaluate the variety of social settings to
  test reasoning and strategic behavior of AI agents. We formulate the benchmark
  SPIN-Bench by systematically varying action spaces, state complexity, and the
  number of interacting agents to simulate a variety of social settings where
  success depends on not only methodical and step-wise decision making, but also
  conceptual inference of other (adversarial or cooperative) participants.
  Our experiments reveal that while contemporary LLMs handle basic fact
  retrieval and short-range planning reasonably well, they encounter
  significant performance bottlenecks in tasks requiring deep multi-hop
  reasoning over large state spaces and socially adept coordination under
  uncertainty. We envision SPIN-Bench as a catalyst for future research on robust
  multi-agent planning, social reasoning, and human--AI teaming.
paper_authors._id: 67d8e49df8b0e148f609c3a6
paper_authors.name: Jianzhu Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:07:54.531Z
paper_authors._id: 660dc90fab11c96e148f95f1
paper_authors.avatarUrl: /avatars/b59fe4b7c16602ea33887a7517ebacb0.svg
paper_authors.fullname: Albert
paper_authors.type: user
paper_authors.user: yyyyyyjjjjzzz
paper_authors._id: 67d8e49df8b0e148f609c3a7
paper_authors.name: Kevin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T14:58:01.093Z
paper_authors._id: 66b979d9088299999ab714b7
paper_authors.avatarUrl: /avatars/28f941da791e51788a4d8e2321c00ded.svg
paper_authors.fullname: Kevin Wang
paper_authors.type: user
paper_authors.user: Huanyu123
paper_authors._id: 67d8e49df8b0e148f609c3a8
paper_authors.name: Ryan Hsieh
paper_authors._id: 67d8e49df8b0e148f609c3a9
paper_authors.name: Haisu Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:07:43.752Z
paper_authors._id: 672a86150957c590b21e49a2
paper_authors.avatarUrl: /avatars/38b12f09af1ad994615b119cb3928404.svg
paper_authors.fullname: Monica Zhou
paper_authors.type: user
paper_authors.user: monicazhou1568
paper_authors._id: 67d8e49df8b0e148f609c3aa
paper_authors.name: Tianqing Zou
paper_authors._id: 67d8e49df8b0e148f609c3ab
paper_authors.name: Zerui Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:07:56.447Z
paper_authors._id: 67d8e5f041fc44bb16e5468c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jUCe0rf7Z-cF51OC5Iczn.png
paper_authors.fullname: Zerui Cheng
paper_authors.type: user
paper_authors.user: ZeruiCheng
paper_authors._id: 67d8e49df8b0e148f609c3ac
paper_authors.name: Zhangyang Wang
paper_authors._id: 67d8e49df8b0e148f609c3ad
paper_authors.name: Pramod Viswanath
paper_title: SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?
paper_summary: Reasoning and strategic behavior in social interactions is a hallmark
  of intelligence. This form of reasoning is significantly more sophisticated
  than isolated planning or reasoning tasks in static settings (e.g., math
  problem solving). In this paper, we present Strategic Planning,
  Interaction, and Negotiation (SPIN-Bench), a new multi-domain
  evaluation designed to measure the intelligence of strategic planning
  and social reasoning. While many existing benchmarks focus on narrow
  planning or single-agent reasoning, SPIN-Bench combines classical PDDL tasks,
  competitive board games, cooperative card games, and multi-agent negotiation
  scenarios in one unified framework. The framework includes both a benchmark as
  well as an arena to simulate and evaluate the variety of social settings to
  test reasoning and strategic behavior of AI agents. We formulate the benchmark
  SPIN-Bench by systematically varying action spaces, state complexity, and the
  number of interacting agents to simulate a variety of social settings where
  success depends on not only methodical and step-wise decision making, but also
  conceptual inference of other (adversarial or cooperative) participants.
  Our experiments reveal that while contemporary LLMs handle basic fact
  retrieval and short-range planning reasonably well, they encounter
  significant performance bottlenecks in tasks requiring deep multi-hop
  reasoning over large state spaces and socially adept coordination under
  uncertainty. We envision SPIN-Bench as a catalyst for future research on robust
  multi-agent planning, social reasoning, and human--AI teaming.
paper_ai_keywords: Strategic Planning Interaction Negotiation SPIN-Bench multi-domain evaluation strategic planning social reasoning PDDL tasks competitive board games cooperative card games multi-agent negotiation scenarios action spaces state complexity conceptual inference basic fact retrieval short-range planning deep multi-hop reasoning socially adept coordination multi-agent planning human--AI teaming