SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via
  Multi-Agent Multi-Turn Reinforcement Learning
Recent advances in reinforcement learning have shown that language models can
develop sophisticated reasoning through training on tasks with verifiable
rewards, but these approaches depend on human-curated problem-answer pairs and
domain-specific reward engineering. We introduce SPIRAL, a self-play framework
where models learn by playing multi-turn, zero-sum games against continuously
improving versions of themselves, eliminating the need for human supervision.
Through self-play, SPIRAL generates an infinite curriculum of progressively
challenging problems as models must constantly adapt to stronger opponents. To
enable this self-play training at scale, We implement a fully online,
multi-turn, multi-agent reinforcement learning system for LLMs and propose
role-conditioned advantage estimation (RAE) to stabilize multi-agent training.
Using SPIRAL, self-play on zero-sum games produces reasoning capabilities that
transfer broadly. Training Qwen3-4B-Base on Kuhn Poker alone achieves 8.6%
improvement on math and 8.4% on general reasoning, outperforming SFT on 25,000
expert game trajectories. Analysis reveals that this transfer occurs through
three cognitive patterns: systematic decomposition, expected value calculation,
and case-by-case analysis. Multi-game training (TicTacToe, Kuhn Poker, Simple
Negotiation) further enhances performance as each game develops distinct
reasoning strengths. Applying SPIRAL to a strong reasoning model
(DeepSeek-R1-Distill-Qwen-7B) can still lead to 2.0% average improvement. These
results demonstrate that zero-sum games naturally develop transferable
reasoning capabilities, highlighting a promising direction for autonomous
reasoning development.
68634850588cea0da970c892
Bo Liu
claimed_verified
2025-07-01T10:13:46.740Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
user
Benjamin-eecs
68634850588cea0da970c893
Leon Guertler
claimed_verified
2025-07-02T07:40:58.689Z
628b671f8fb67b90658613f7
https://cdn-avatars.huggingface.co/v1/production/uploads/1653303027789-noauth.jpeg
Leon Guertler
user
LeonGuertler
68634850588cea0da970c894
Simon Yu
claimed_verified
2025-07-01T10:13:44.863Z
636681feaa6a4af6073ba73e
https://cdn-avatars.huggingface.co/v1/production/uploads/636681feaa6a4af6073ba73e/u_0moYNu6as9Sszp-ej95.png
Simon Yu
user
simonycl
68634850588cea0da970c895
Zichen Liu
claimed_verified
2025-07-01T10:13:42.588Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
68634850588cea0da970c896
Penghui Qi
68634850588cea0da970c897
Daniel Balcells
claimed_verified
2025-07-16T08:16:06.283Z
65722278a41546bf7cf3b009
https://cdn-avatars.huggingface.co/v1/production/uploads/65722278a41546bf7cf3b009/aGfBPWpdIyCxQh7mPF3wb.jpeg
Dani Balcells
user
danibalcells
68634850588cea0da970c898
Mickel Liu
68634850588cea0da970c899
Cheston Tan
68634850588cea0da970c89a
Weiyan Shi
68634850588cea0da970c89b
Min Lin
68634850588cea0da970c89c
Wee Sun Lee
68634850588cea0da970c89d
Natasha Jaques
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via
  Multi-Agent Multi-Turn Reinforcement Learning
Recent advances in reinforcement learning have shown that language models can
develop sophisticated reasoning through training on tasks with verifiable
rewards, but these approaches depend on human-curated problem-answer pairs and
domain-specific reward engineering. We introduce SPIRAL, a self-play framework
where models learn by playing multi-turn, zero-sum games against continuously
improving versions of themselves, eliminating the need for human supervision.
Through self-play, SPIRAL generates an infinite curriculum of progressively
challenging problems as models must constantly adapt to stronger opponents. To
enable this self-play training at scale, We implement a fully online,
multi-turn, multi-agent reinforcement learning system for LLMs and propose
role-conditioned advantage estimation (RAE) to stabilize multi-agent training.
Using SPIRAL, self-play on zero-sum games produces reasoning capabilities that
transfer broadly. Training Qwen3-4B-Base on Kuhn Poker alone achieves 8.6%
improvement on math and 8.4% on general reasoning, outperforming SFT on 25,000
expert game trajectories. Analysis reveals that this transfer occurs through
three cognitive patterns: systematic decomposition, expected value calculation,
and case-by-case analysis. Multi-game training (TicTacToe, Kuhn Poker, Simple
Negotiation) further enhances performance as each game develops distinct
reasoning strengths. Applying SPIRAL to a strong reasoning model
(DeepSeek-R1-Distill-Qwen-7B) can still lead to 2.0% average improvement. These
results demonstrate that zero-sum games naturally develop transferable
reasoning capabilities, highlighting a promising direction for autonomous
reasoning development.
Self-play in zero-sum games using SPIRAL framework enhances reasoning capabilities in language models through continuous adaptation and transfer learning.
reinforcement learning self-play multi-turn zero-sum games role-conditioned advantage estimation RAE multi-agent reinforcement learning Kuhn Poker TicTacToe Simple Negotiation DeepSeek-R1-Distill-Qwen-7B