title: Efficient Process Reward Model Training via Active Learning
summary: Process Reward Models (PRMs) provide step-level supervision to large language
  models (LLMs), but scaling up training data annotation remains challenging for
  both humans and LLMs. To address this limitation, we propose an active learning
  approach, ActPRM, which proactively selects the most uncertain samples for
  training, substantially reducing labeling costs. During training, we use the
  PRM to estimate uncertainty after the forward pass, retaining only highly
  uncertain data. A capable yet costly reasoning model then labels this data.
  Then we compute the loss with respect to the labels and update the PRM's
  weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active
  learning setting, demonstrating that ActPRM reduces 50% annotation, but
  achieving the comparable or even better performance. Beyond annotation
  efficiency, we further advance the actively trained PRM by filtering over 1M+
  math reasoning trajectories with ActPRM, retaining 60% of the data. A
  subsequent training on this selected dataset yields a new state-of-the-art
  (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same
  sized models.
paper_authors._id: 67ff1df03b42083b37219456
paper_authors.name: Keyu Duan
paper_authors._id: 67ff1df03b42083b37219457
paper_authors.name: Zichen Liu
paper_authors._id: 67ff1df03b42083b37219458
paper_authors.name: Xin Mao
paper_authors._id: 67ff1df03b42083b37219459
paper_authors.name: Tianyu Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:03:43.946Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 67ff1df03b42083b3721945a
paper_authors.name: Changyu Chen
paper_authors._id: 67ff1df03b42083b3721945b
paper_authors.name: Qiguang Chen
paper_authors._id: 67ff1df03b42083b3721945c
paper_authors.name: Michael Qizhe Shieh
paper_authors._id: 67ff1df03b42083b3721945d
paper_authors.name: Longxu Dou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T08:43:00.444Z
paper_authors._id: 6214e4ee1e35c843d42d1f88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
paper_authors.fullname: Longxu Dou
paper_authors.type: user
paper_authors.user: dreamerdeo
paper_title: Efficient Process Reward Model Training via Active Learning
paper_summary: Process Reward Models (PRMs) provide step-level supervision to large language
  models (LLMs), but scaling up training data annotation remains challenging for
  both humans and LLMs. To address this limitation, we propose an active learning
  approach, ActPRM, which proactively selects the most uncertain samples for
  training, substantially reducing labeling costs. During training, we use the
  PRM to estimate uncertainty after the forward pass, retaining only highly
  uncertain data. A capable yet costly reasoning model then labels this data.
  Then we compute the loss with respect to the labels and update the PRM's
  weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active
  learning setting, demonstrating that ActPRM reduces 50% annotation, but
  achieving the comparable or even better performance. Beyond annotation
  efficiency, we further advance the actively trained PRM by filtering over 1M+
  math reasoning trajectories with ActPRM, retaining 60% of the data. A
  subsequent training on this selected dataset yields a new state-of-the-art
  (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same
  sized models.
paper_ai_summary: An active learning method named ActPRM selects uncertain data for training Process Reward Models, reducing annotation costs while maintaining or improving performance.
paper_ai_keywords: active learning Process Reward Models PRMs large language models LLMs forward pass pool-based active learning ProcessBench PRMBench