title: GenPRM: Scaling Test-Time Compute of Process Reward Models via
    Generative Reasoning
summary: Recent advancements in Large Language Models (LLMs) have shown that it is
  promising to utilize Process Reward Models (PRMs) as verifiers to enhance the
  performance of LLMs. However, current PRMs face three key challenges: (1)
  limited process supervision and generalization capabilities, (2) dependence on
  scalar value prediction without leveraging the generative abilities of LLMs,
  and (3) inability to scale the test-time compute of PRMs. In this work, we
  introduce GenPRM, a generative process reward model that performs explicit
  Chain-of-Thought (CoT) reasoning with code verification before providing
  judgment for each reasoning step. To obtain high-quality process supervision
  labels and rationale data, we propose Relative Progress Estimation (RPE) and a
  rationale synthesis framework that incorporates code verification. Experimental
  results on ProcessBench and several mathematical reasoning tasks show that
  GenPRM significantly outperforms prior PRMs with only 23K training data from
  MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and
  a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally,
  GenPRM demonstrates strong abilities to serve as a critic model for policy
  model refinement. This work establishes a new paradigm for process supervision
  that bridges the gap between PRMs and critic models in LLMs. Our code, model,
  and data will be available in https://ryanliu112.github.io/GenPRM.
paper_authors._id: 67ef62342a18e60aeee0ea02
paper_authors.name: Jian Zhao
paper_authors._id: 67ef62342a18e60aeee0ea03
paper_authors.name: Runze Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:09.923Z
paper_authors._id: 667187ba9ab144eb3ac43a1b
paper_authors.avatarUrl: /avatars/db5558aa1c5160b9aee8b58573271959.svg
paper_authors.fullname: Runze Liu
paper_authors.type: user
paper_authors.user: RyanLiu112
paper_authors._id: 67ef62342a18e60aeee0ea04
paper_authors.name: Kaiyan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:25:22.709Z
paper_authors._id: 60bc94cd85a3ab33829b6211
paper_authors.avatarUrl: /avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
paper_authors.fullname: Kaiyan Zhang
paper_authors.type: user
paper_authors.user: iseesaw
paper_authors._id: 67ef62342a18e60aeee0ea05
paper_authors.name: Zhimu Zhou
paper_authors._id: 67ef62342a18e60aeee0ea06
paper_authors.name: Junqi Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:25:38.624Z
paper_authors._id: 67ab05fe4c6ca2d5db4c0c52
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QpGUNDkeuKjX71s2GXlXF.png
paper_authors.fullname: Junqi Gao
paper_authors.type: user
paper_authors.user: ChetKao
paper_authors._id: 67ef62342a18e60aeee0ea07
paper_authors.name: Dong Li
paper_authors._id: 67ef62342a18e60aeee0ea08
paper_authors.name: Jiafei Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:25:45.394Z
paper_authors._id: 6562db314e8918182da42706
paper_authors.avatarUrl: /avatars/b113bbbb496bf4dac254f0e840f08e10.svg
paper_authors.fullname: Jiafei Lyu
paper_authors.type: user
paper_authors.user: dmux
paper_authors._id: 67ef62342a18e60aeee0ea09
paper_authors.name: Zhouyi Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:26:00.763Z
paper_authors._id: 65b34c5785b6c2144807db37
paper_authors.avatarUrl: /avatars/4c1cb03cda250d4ec760ebf7815a3bce.svg
paper_authors.fullname: Qianzhouyi
paper_authors.type: user
paper_authors.user: Saputello
paper_authors._id: 67ef62342a18e60aeee0ea0a
paper_authors.name: Biqing Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:26:06.517Z
paper_authors._id: 645d9c3058f9ee315148116d
paper_authors.avatarUrl: /avatars/165e18f27b5a50738bf1d22857118478.svg
paper_authors.fullname: Biqing Qi
paper_authors.type: user
paper_authors.user: jackqi7
paper_authors._id: 67ef62342a18e60aeee0ea0b
paper_authors.name: Xiu Li
paper_authors._id: 67ef62342a18e60aeee0ea0c
paper_authors.name: Bowen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:26:14.312Z
paper_authors._id: 669f614b59adf5b56e05bce3
paper_authors.avatarUrl: /avatars/ffd4189efbceb0e63a03db273065a44b.svg
paper_authors.fullname: BowenZhou
paper_authors.type: user
paper_authors.user: bowenZhou
paper_title: GenPRM: Scaling Test-Time Compute of Process Reward Models via
    Generative Reasoning
paper_summary: Recent advancements in Large Language Models (LLMs) have shown that it is
  promising to utilize Process Reward Models (PRMs) as verifiers to enhance the
  performance of LLMs. However, current PRMs face three key challenges: (1)
  limited process supervision and generalization capabilities, (2) dependence on
  scalar value prediction without leveraging the generative abilities of LLMs,
  and (3) inability to scale the test-time compute of PRMs. In this work, we
  introduce GenPRM, a generative process reward model that performs explicit
  Chain-of-Thought (CoT) reasoning with code verification before providing
  judgment for each reasoning step. To obtain high-quality process supervision
  labels and rationale data, we propose Relative Progress Estimation (RPE) and a
  rationale synthesis framework that incorporates code verification. Experimental
  results on ProcessBench and several mathematical reasoning tasks show that
  GenPRM significantly outperforms prior PRMs with only 23K training data from
  MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and
  a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally,
  GenPRM demonstrates strong abilities to serve as a critic model for policy
  model refinement. This work establishes a new paradigm for process supervision
  that bridges the gap between PRMs and critic models in LLMs. Our code, model,
  and data will be available in https://ryanliu112.github.io/GenPRM.
paper_ai_summary: GenPRM, a generative process reward model with Chain-of-Thought reasoning and code verification, outperforms existing PRMs and critic models in process supervision tasks with less data and computational resources.
paper_ai_keywords: Large Language Models Process Reward Models Chain-of-Thought Relative Progress Estimation rationale synthesis GenPRM ProcessBench mathematical reasoning tasks critic model policy model refinement