Process Reward Models That Think
Step-by-step verifiers -- also known as process reward models (PRMs) -- are a
key ingredient for test-time scaling. PRMs require step-level supervision,
making them expensive to train. This work aims to build data-efficient PRMs as
verbalized step-wise reward models that verify every step in the solution by
generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long
CoT verifier fine-tuned on orders of magnitude fewer process labels than those
required by discriminative PRMs. Our approach capitalizes on the inherent
reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and
discriminative verifiers -- using only 1% of the process labels in PRM800K --
across several challenging benchmarks. Specifically, ThinkPRM beats the
baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and
reward-guided search. In an out-of-domain evaluation on a subset of
GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers
trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the
same token budget, ThinkPRM scales up verification compute more effectively
compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of
ProcessBench. Our work highlights the value of generative, long CoT PRMs that
can scale test-time compute for verification while requiring minimal
supervision for training. Our code, data, and models will be released at
https://github.com/mukhal/thinkprm.
680a831b49e5373a4f6c2cfe
Muhammad Khalifa
claimed_verified
2025-06-02T07:50:38.306Z
5f350fe67e5835433862161b
/avatars/8aecfdea5202968b1099412800a152e0.svg
Muhammad Khalifa
user
mkhalifa
680a831b49e5373a4f6c2cff
Rishabh Agarwal
680a831b49e5373a4f6c2d00
Lajanugen Logeswaran
680a831b49e5373a4f6c2d01
Jaekyeom Kim
680a831b49e5373a4f6c2d02
Hao Peng
680a831b49e5373a4f6c2d03
Moontae Lee
680a831b49e5373a4f6c2d04
Honglak Lee
680a831b49e5373a4f6c2d05
Lu Wang
Process Reward Models That Think
Step-by-step verifiers -- also known as process reward models (PRMs) -- are a
key ingredient for test-time scaling. PRMs require step-level supervision,
making them expensive to train. This work aims to build data-efficient PRMs as
verbalized step-wise reward models that verify every step in the solution by
generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long
CoT verifier fine-tuned on orders of magnitude fewer process labels than those
required by discriminative PRMs. Our approach capitalizes on the inherent
reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and
discriminative verifiers -- using only 1% of the process labels in PRM800K --
across several challenging benchmarks. Specifically, ThinkPRM beats the
baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and
reward-guided search. In an out-of-domain evaluation on a subset of
GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers
trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the
same token budget, ThinkPRM scales up verification compute more effectively
compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of
ProcessBench. Our work highlights the value of generative, long CoT PRMs that
can scale test-time compute for verification while requiring minimal
supervision for training. Our code, data, and models will be released at
https://github.com/mukhal/thinkprm.
ThinkPRM, a long chain-of-thought verifier, uses minimal supervision to outperform discriminative PRMs and LLM-as-a-Judge across various benchmarks and token budgets.
step-by-step verifiers process reward models (PRMs) step-level supervision long chain-of-thought (CoT) ThinkPRM LLM-as-a-Judge discriminative verifiers ProcessBench MATH-500 AIME '24 GPQA-Diamond LiveCodeBench verification compute token budget