title: Process Reward Models That Think
summary: Step-by-step verifiers -- also known as process reward models (PRMs) -- are a
  key ingredient for test-time scaling. PRMs require step-level supervision,
  making them expensive to train. This work aims to build data-efficient PRMs as
  verbalized step-wise reward models that verify every step in the solution by
  generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long
  CoT verifier fine-tuned on orders of magnitude fewer process labels than those
  required by discriminative PRMs. Our approach capitalizes on the inherent
  reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and
  discriminative verifiers -- using only 1% of the process labels in PRM800K --
  across several challenging benchmarks. Specifically, ThinkPRM beats the
  baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and
  reward-guided search. In an out-of-domain evaluation on a subset of
  GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers
  trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the
  same token budget, ThinkPRM scales up verification compute more effectively
  compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of
  ProcessBench. Our work highlights the value of generative, long CoT PRMs that
  can scale test-time compute for verification while requiring minimal
  supervision for training. Our code, data, and models will be released at
  https://github.com/mukhal/thinkprm.
paper_authors._id: 680a831b49e5373a4f6c2cfe
paper_authors.name: Muhammad Khalifa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:50:38.306Z
paper_authors._id: 5f350fe67e5835433862161b
paper_authors.avatarUrl: /avatars/8aecfdea5202968b1099412800a152e0.svg
paper_authors.fullname: Muhammad Khalifa
paper_authors.type: user
paper_authors.user: mkhalifa
paper_authors._id: 680a831b49e5373a4f6c2cff
paper_authors.name: Rishabh Agarwal
paper_authors._id: 680a831b49e5373a4f6c2d00
paper_authors.name: Lajanugen Logeswaran
paper_authors._id: 680a831b49e5373a4f6c2d01
paper_authors.name: Jaekyeom Kim
paper_authors._id: 680a831b49e5373a4f6c2d02
paper_authors.name: Hao Peng
paper_authors._id: 680a831b49e5373a4f6c2d03
paper_authors.name: Moontae Lee
paper_authors._id: 680a831b49e5373a4f6c2d04
paper_authors.name: Honglak Lee
paper_authors._id: 680a831b49e5373a4f6c2d05
paper_authors.name: Lu Wang
paper_title: Process Reward Models That Think
paper_summary: Step-by-step verifiers -- also known as process reward models (PRMs) -- are a
  key ingredient for test-time scaling. PRMs require step-level supervision,
  making them expensive to train. This work aims to build data-efficient PRMs as
  verbalized step-wise reward models that verify every step in the solution by
  generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long
  CoT verifier fine-tuned on orders of magnitude fewer process labels than those
  required by discriminative PRMs. Our approach capitalizes on the inherent
  reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and
  discriminative verifiers -- using only 1% of the process labels in PRM800K --
  across several challenging benchmarks. Specifically, ThinkPRM beats the
  baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and
  reward-guided search. In an out-of-domain evaluation on a subset of
  GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers
  trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the
  same token budget, ThinkPRM scales up verification compute more effectively
  compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of
  ProcessBench. Our work highlights the value of generative, long CoT PRMs that
  can scale test-time compute for verification while requiring minimal
  supervision for training. Our code, data, and models will be released at
  https://github.com/mukhal/thinkprm.
paper_ai_summary: ThinkPRM, a long chain-of-thought verifier, uses minimal supervision to outperform discriminative PRMs and LLM-as-a-Judge across various benchmarks and token budgets.
paper_ai_keywords: step-by-step verifiers process reward models (PRMs) step-level supervision long chain-of-thought (CoT) ThinkPRM LLM-as-a-Judge discriminative verifiers ProcessBench MATH-500 AIME '24 GPQA-Diamond LiveCodeBench verification compute token budget