Efficient Process Reward Model Training via Active Learning
Process Reward Models (PRMs) provide step-level supervision to large language
models (LLMs), but scaling up training data annotation remains challenging for
both humans and LLMs. To address this limitation, we propose an active learning
approach, ActPRM, which proactively selects the most uncertain samples for
training, substantially reducing labeling costs. During training, we use the
PRM to estimate uncertainty after the forward pass, retaining only highly
uncertain data. A capable yet costly reasoning model then labels this data.
Then we compute the loss with respect to the labels and update the PRM's
weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active
learning setting, demonstrating that ActPRM reduces 50% annotation, but
achieving the comparable or even better performance. Beyond annotation
efficiency, we further advance the actively trained PRM by filtering over 1M+
math reasoning trajectories with ActPRM, retaining 60% of the data. A
subsequent training on this selected dataset yields a new state-of-the-art
(SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same
sized models.
67ff1df03b42083b37219456
Keyu Duan
67ff1df03b42083b37219457
Zichen Liu
67ff1df03b42083b37219458
Xin Mao
67ff1df03b42083b37219459
Tianyu Pang
claimed_verified
2025-04-20T15:03:43.946Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
67ff1df03b42083b3721945a
Changyu Chen
67ff1df03b42083b3721945b
Qiguang Chen
67ff1df03b42083b3721945c
Michael Qizhe Shieh
67ff1df03b42083b3721945d
Longxu Dou
claimed_verified
2025-04-16T08:43:00.444Z
6214e4ee1e35c843d42d1f88
https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
Longxu Dou
user
dreamerdeo
Efficient Process Reward Model Training via Active Learning
Process Reward Models (PRMs) provide step-level supervision to large language
models (LLMs), but scaling up training data annotation remains challenging for
both humans and LLMs. To address this limitation, we propose an active learning
approach, ActPRM, which proactively selects the most uncertain samples for
training, substantially reducing labeling costs. During training, we use the
PRM to estimate uncertainty after the forward pass, retaining only highly
uncertain data. A capable yet costly reasoning model then labels this data.
Then we compute the loss with respect to the labels and update the PRM's
weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active
learning setting, demonstrating that ActPRM reduces 50% annotation, but
achieving the comparable or even better performance. Beyond annotation
efficiency, we further advance the actively trained PRM by filtering over 1M+
math reasoning trajectories with ActPRM, retaining 60% of the data. A
subsequent training on this selected dataset yields a new state-of-the-art
(SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same
sized models.
An active learning method named ActPRM selects uncertain data for training Process Reward Models, reducing annotation costs while maintaining or improving performance.
active learning Process Reward Models PRMs large language models LLMs forward pass pool-based active learning ProcessBench PRMBench