title: RM-R1: Reward Modeling as Reasoning
summary: Reward modeling is essential for aligning large language models (LLMs) with
  human preferences, especially through reinforcement learning from human
  feedback (RLHF). To provide accurate reward signals, a reward model (RM) should
  stimulate deep thinking and conduct interpretable reasoning before assigning a
  score or a judgment. However, existing RMs either produce opaque scalar scores
  or directly generate the prediction of a preferred answer, making them struggle
  to integrate natural language critiques, thus lacking interpretability.
  Inspired by recent advances of long chain-of-thought (CoT) on
  reasoning-intensive tasks, we hypothesize and validate that integrating
  reasoning capabilities into reward modeling significantly enhances RM's
  interpretability and performance. In this work, we introduce a new class of
  generative reward models -- Reasoning Reward Models (ReasRMs) -- which
  formulate reward modeling as a reasoning task. We propose a reasoning-oriented
  training pipeline and train a family of ReasRMs, RM-R1. The training consists
  of two key stages: (1) distillation of high-quality reasoning chains and (2)
  reinforcement learning with verifiable rewards. RM-R1 improves LLM rollouts by
  self-generating reasoning traces or chat-specific rubrics and evaluating
  candidate responses against them. Empirically, our models achieve
  state-of-the-art or near state-of-the-art performance of generative RMs across
  multiple comprehensive reward model benchmarks, outperforming much larger
  open-weight models (e.g., Llama3.1-405B) and proprietary ones (e.g., GPT-4o) by
  up to 13.8%. Beyond final performance, we perform thorough empirical analysis
  to understand the key ingredients of successful ReasRM training. To facilitate
  future research, we release six ReasRM models along with code and data at
  https://github.com/RM-R1-UIUC/RM-R1.
paper_authors._id: 681988d6d6a5fee26b52ac28
paper_authors.name: Xiusi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:47:11.654Z
paper_authors._id: 6270ff726417aed8a7340c8b
paper_authors.avatarUrl: /avatars/3f14913c55cc4fc78678ac43fb603e80.svg
paper_authors.fullname: Xiusi Chen
paper_authors.type: user
paper_authors.user: XtremSup
paper_authors._id: 681988d6d6a5fee26b52ac29
paper_authors.name: Gaotang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:33:13.258Z
paper_authors._id: 654d784d71a30c4bca09a319
paper_authors.avatarUrl: /avatars/ab9f93122903ccd662267232bab30ad8.svg
paper_authors.fullname: Gaotang Li
paper_authors.type: user
paper_authors.user: gaotang
paper_authors._id: 681988d6d6a5fee26b52ac2a
paper_authors.name: Ziqi Wang
paper_authors._id: 681988d6d6a5fee26b52ac2b
paper_authors.name: Bowen Jin
paper_authors._id: 681988d6d6a5fee26b52ac2c
paper_authors.name: Cheng Qian
paper_authors._id: 681988d6d6a5fee26b52ac2d
paper_authors.name: Yu Wang
paper_authors._id: 681988d6d6a5fee26b52ac2e
paper_authors.name: Hongru Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:33:11.136Z
paper_authors._id: 65f906e5c3dbdcae83ff7aac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
paper_authors.fullname: Hongru Wang
paper_authors.type: user
paper_authors.user: Merlin-Hongru
paper_authors._id: 681988d6d6a5fee26b52ac2f
paper_authors.name: Yu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:50:17.864Z
paper_authors._id: 64b78f80b5d293ec95a3f694
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b78f80b5d293ec95a3f694/3rKg-m2i6ulj_aYhldvU8.png
paper_authors.fullname: Yu Zhang
paper_authors.type: user
paper_authors.user: yuz9yuz
paper_authors._id: 681988d6d6a5fee26b52ac30
paper_authors.name: Denghui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:48:00.793Z
paper_authors._id: 66285acb73af5913c6bbf1ec
paper_authors.avatarUrl: /avatars/8969e3a6ae2dcc0b1c49768fd044b9e0.svg
paper_authors.fullname: Denghui Zhang
paper_authors.type: user
paper_authors.user: zhangdenghui123
paper_authors._id: 681988d6d6a5fee26b52ac31
paper_authors.name: Tong Zhang
paper_authors._id: 681988d6d6a5fee26b52ac32
paper_authors.name: Hanghang Tong
paper_authors._id: 681988d6d6a5fee26b52ac33
paper_authors.name: Heng Ji
paper_title: RM-R1: Reward Modeling as Reasoning
paper_summary: Reward modeling is essential for aligning large language models (LLMs) with
  human preferences, especially through reinforcement learning from human
  feedback (RLHF). To provide accurate reward signals, a reward model (RM) should
  stimulate deep thinking and conduct interpretable reasoning before assigning a
  score or a judgment. However, existing RMs either produce opaque scalar scores
  or directly generate the prediction of a preferred answer, making them struggle
  to integrate natural language critiques, thus lacking interpretability.
  Inspired by recent advances of long chain-of-thought (CoT) on
  reasoning-intensive tasks, we hypothesize and validate that integrating
  reasoning capabilities into reward modeling significantly enhances RM's
  interpretability and performance. In this work, we introduce a new class of
  generative reward models -- Reasoning Reward Models (ReasRMs) -- which
  formulate reward modeling as a reasoning task. We propose a reasoning-oriented
  training pipeline and train a family of ReasRMs, RM-R1. The training consists
  of two key stages: (1) distillation of high-quality reasoning chains and (2)
  reinforcement learning with verifiable rewards. RM-R1 improves LLM rollouts by
  self-generating reasoning traces or chat-specific rubrics and evaluating
  candidate responses against them. Empirically, our models achieve
  state-of-the-art or near state-of-the-art performance of generative RMs across
  multiple comprehensive reward model benchmarks, outperforming much larger
  open-weight models (e.g., Llama3.1-405B) and proprietary ones (e.g., GPT-4o) by
  up to 13.8%. Beyond final performance, we perform thorough empirical analysis
  to understand the key ingredients of successful ReasRM training. To facilitate
  future research, we release six ReasRM models along with code and data at
  https://github.com/RM-R1-UIUC/RM-R1.
paper_ai_summary: Reasoning Reward Models (ReasRMs) enhance reward modeling for large language models by integrating reasoning tasks, improving interpretability and performance.
paper_ai_keywords: reward modeling reinforcement learning from human feedback (RLHF) reward model (RM) long chain-of-thought (CoT) reasoning capabilities Reasoning Reward Models (ReasRMs) reasoning-oriented training pipeline high-quality reasoning chains reinforcement learning with verifiable rewards LLM rollouts reasoning traces chat-specific rubrics state-of-the-art performance reward model benchmarks Llama3.1-405B GPT-4o