title: SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
summary: Evaluating the step-by-step reliability of large language model (LLM)
  reasoning, such as Chain-of-Thought, remains challenging due to the difficulty
  and cost of obtaining high-quality step-level supervision. In this paper, we
  introduce Self-Play Critic (SPC), a novel approach where a critic model evolves
  its ability to assess reasoning steps through adversarial self-play games,
  eliminating the need for manual step-level annotation. SPC involves fine-tuning
  two copies of a base model to play two roles, namely a "sneaky generator" that
  deliberately produces erroneous steps designed to be difficult to detect, and a
  "critic" that analyzes the correctness of reasoning steps. These two models
  engage in an adversarial game in which the generator aims to fool the critic,
  while the critic model seeks to identify the generator's errors. Using
  reinforcement learning based on the game outcomes, the models iteratively
  improve; the winner of each confrontation receives a positive reward and the
  loser receives a negative reward, driving continuous self-evolution.
  Experiments on three reasoning process benchmarks (ProcessBench, PRM800K,
  DeltaBench) demonstrate that our SPC progressively enhances its error detection
  capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and
  surpasses strong baselines, including distilled R1 model. Furthermore, applying
  SPC to guide the test-time search of diverse LLMs significantly improves their
  mathematical reasoning performance on MATH500 and AIME2024, outperforming
  state-of-the-art process reward models.
paper_authors._id: 6810799e10b86ba322c27fb8
paper_authors.name: Jiaqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-29T07:58:59.273Z
paper_authors._id: 61b859ddbdf1fac5ed499992
paper_authors.avatarUrl: /avatars/2387fb9b8a46840bfc75248462f0a410.svg
paper_authors.fullname: Jiaqi Chen
paper_authors.type: user
paper_authors.user: judge
paper_authors._id: 6810799e10b86ba322c27fb9
paper_authors.name: Bang Zhang
paper_authors._id: 6810799e10b86ba322c27fba
paper_authors.name: Ruotian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:24:07.283Z
paper_authors._id: 648294b2eb4befee378951c1
paper_authors.avatarUrl: /avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
paper_authors.fullname: Ruotian Ma
paper_authors.type: user
paper_authors.user: vvibt
paper_authors._id: 6810799e10b86ba322c27fbb
paper_authors.name: Peisong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:24:12.917Z
paper_authors._id: 626f98528a894872cfbf620c
paper_authors.avatarUrl: /avatars/fe31d20313e6ca85e96bc249424c5383.svg
paper_authors.fullname: Peisong Wang
paper_authors.type: user
paper_authors.user: duke1852022
paper_authors._id: 6810799e10b86ba322c27fbc
paper_authors.name: Xiaodan Liang
paper_authors._id: 6810799e10b86ba322c27fbd
paper_authors.name: Zhaopeng Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:24:26.048Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_authors._id: 6810799e10b86ba322c27fbe
paper_authors.name: Xiaolong Li
paper_authors._id: 6810799e10b86ba322c27fbf
paper_authors.name: Kwan-Yee K. Wong
paper_title: SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
paper_summary: Evaluating the step-by-step reliability of large language model (LLM)
  reasoning, such as Chain-of-Thought, remains challenging due to the difficulty
  and cost of obtaining high-quality step-level supervision. In this paper, we
  introduce Self-Play Critic (SPC), a novel approach where a critic model evolves
  its ability to assess reasoning steps through adversarial self-play games,
  eliminating the need for manual step-level annotation. SPC involves fine-tuning
  two copies of a base model to play two roles, namely a "sneaky generator" that
  deliberately produces erroneous steps designed to be difficult to detect, and a
  "critic" that analyzes the correctness of reasoning steps. These two models
  engage in an adversarial game in which the generator aims to fool the critic,
  while the critic model seeks to identify the generator's errors. Using
  reinforcement learning based on the game outcomes, the models iteratively
  improve; the winner of each confrontation receives a positive reward and the
  loser receives a negative reward, driving continuous self-evolution.
  Experiments on three reasoning process benchmarks (ProcessBench, PRM800K,
  DeltaBench) demonstrate that our SPC progressively enhances its error detection
  capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and
  surpasses strong baselines, including distilled R1 model. Furthermore, applying
  SPC to guide the test-time search of diverse LLMs significantly improves their
  mathematical reasoning performance on MATH500 and AIME2024, outperforming
  state-of-the-art process reward models.
paper_ai_summary: Self-Play Critic (SPC) enhances LLM reasoning reliability by using adversarial self-play to improve error detection and reasoning performance without requiring manual step-level annotation.
paper_ai_keywords: Self-Play Critic Chain-of-Thought adversarial self-play reinforcement learning reasoning steps error detection ProcessBench PRM800K DeltaBench mathematical reasoning MATH500 AIME2024