title: The Lessons of Developing Process Reward Models in Mathematical
    Reasoning
summary: Process Reward Models (PRMs) emerge as a promising approach for process
  supervision in mathematical reasoning of Large Language Models (LLMs), which
  aim to identify and mitigate intermediate errors in the reasoning processes.
  However, the development of effective PRMs faces significant challenges,
  particularly in data annotation and evaluation methodologies. In this paper,
  through extensive experiments, we demonstrate that commonly used Monte Carlo
  (MC) estimation-based data synthesis for PRMs typically yields inferior
  performance and generalization compared to LLM-as-a-judge and human annotation
  methods. MC estimation relies on completion models to evaluate current-step
  correctness, leading to inaccurate step verification. Furthermore, we identify
  potential biases in conventional Best-of-N (BoN) evaluation strategies for
  PRMs: (1) The unreliable policy models generate responses with correct answers
  but flawed processes, leading to a misalignment between the evaluation criteria
  of BoN and the PRM objectives of process verification. (2) The tolerance of
  PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a
  significant proportion of minimum scores concentrated on the final answer
  steps, revealing the shift from process to outcome-based assessment in BoN
  Optimized PRMs. To address these challenges, we develop a consensus filtering
  mechanism that effectively integrates MC estimation with LLM-as-a-judge and
  advocates a more comprehensive evaluation framework that combines
  response-level and step-level metrics. Based on the mechanisms, we
  significantly improve both model performance and data efficiency in the BoN
  evaluation and the step-wise error identification task. Finally, we release a
  new state-of-the-art PRM that outperforms existing open-source alternatives and
  provides practical guidelines for future research in building process
  supervision models.
paper_authors._id: 6785e601117627fe711f8dd8
paper_authors.name: Zhenru Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:48:52.704Z
paper_authors._id: 64704e973601bb7b06643e98
paper_authors.avatarUrl: /avatars/52e51f4d1be6769e4397b8be2799cf32.svg
paper_authors.fullname: Zhenru Zhang
paper_authors.type: user
paper_authors.user: Zhenru
paper_authors._id: 6785e601117627fe711f8dd9
paper_authors.name: Chujie Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:28:43.769Z
paper_authors._id: 610b70452719facd4ea85e28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
paper_authors.fullname: Chujie Zheng
paper_authors.type: user
paper_authors.user: chujiezheng
paper_authors._id: 6785e601117627fe711f8dda
paper_authors.name: Yangzhen Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:49:01.963Z
paper_authors._id: 671f2a2ab02f88b7e2385105
paper_authors.avatarUrl: /avatars/a97e2f983c8290b72eeb03e2bdcaf085.svg
paper_authors.fullname: Yangzhen Wu
paper_authors.type: user
paper_authors.user: wuyangzhen
paper_authors._id: 6785e601117627fe711f8ddb
paper_authors.name: Beichen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:49:43.861Z
paper_authors._id: 64b93578ee257c3a4cfceed1
paper_authors.avatarUrl: /avatars/e6188562254f75a09b4048b800860016.svg
paper_authors.fullname: Beichen Zhang
paper_authors.type: user
paper_authors.user: BeichenZhang
paper_authors._id: 6785e601117627fe711f8ddc
paper_authors.name: Runji Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:49:56.685Z
paper_authors._id: 649a52e5de0fb7f3f499e583
paper_authors.avatarUrl: /avatars/25f6106fa168ae57ad5cd8ef55c70d31.svg
paper_authors.fullname: Runji Lin
paper_authors.type: user
paper_authors.user: RunjiLin
paper_authors._id: 6785e601117627fe711f8ddd
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:50:12.316Z
paper_authors._id: 6438b43ab2ea24b52ebac2b9
paper_authors.avatarUrl: /avatars/84133cd719a4b1e2f5c1a74178425f86.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: bwy
paper_authors._id: 6785e601117627fe711f8dde
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:50:18.536Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 6785e601117627fe711f8ddf
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:50:29.301Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_authors._id: 6785e601117627fe711f8de0
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:50:35.067Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: The Lessons of Developing Process Reward Models in Mathematical
    Reasoning
paper_summary: Process Reward Models (PRMs) emerge as a promising approach for process
  supervision in mathematical reasoning of Large Language Models (LLMs), which
  aim to identify and mitigate intermediate errors in the reasoning processes.
  However, the development of effective PRMs faces significant challenges,
  particularly in data annotation and evaluation methodologies. In this paper,
  through extensive experiments, we demonstrate that commonly used Monte Carlo
  (MC) estimation-based data synthesis for PRMs typically yields inferior
  performance and generalization compared to LLM-as-a-judge and human annotation
  methods. MC estimation relies on completion models to evaluate current-step
  correctness, leading to inaccurate step verification. Furthermore, we identify
  potential biases in conventional Best-of-N (BoN) evaluation strategies for
  PRMs: (1) The unreliable policy models generate responses with correct answers
  but flawed processes, leading to a misalignment between the evaluation criteria
  of BoN and the PRM objectives of process verification. (2) The tolerance of
  PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a
  significant proportion of minimum scores concentrated on the final answer
  steps, revealing the shift from process to outcome-based assessment in BoN
  Optimized PRMs. To address these challenges, we develop a consensus filtering
  mechanism that effectively integrates MC estimation with LLM-as-a-judge and
  advocates a more comprehensive evaluation framework that combines
  response-level and step-level metrics. Based on the mechanisms, we
  significantly improve both model performance and data efficiency in the BoN
  evaluation and the step-wise error identification task. Finally, we release a
  new state-of-the-art PRM that outperforms existing open-source alternatives and
  provides practical guidelines for future research in building process
  supervision models.
paper_ai_keywords: Process Reward Models (PRMs) Large Language Models (LLMs) Monte Carlo (MC) estimation Best-of-N (BoN) evaluation strategies policy models response-level metrics step-level metrics