RM-R1: Reward Modeling as Reasoning
Reward modeling is essential for aligning large language models (LLMs) with
human preferences, especially through reinforcement learning from human
feedback (RLHF). To provide accurate reward signals, a reward model (RM) should
stimulate deep thinking and conduct interpretable reasoning before assigning a
score or a judgment. However, existing RMs either produce opaque scalar scores
or directly generate the prediction of a preferred answer, making them struggle
to integrate natural language critiques, thus lacking interpretability.
Inspired by recent advances of long chain-of-thought (CoT) on
reasoning-intensive tasks, we hypothesize and validate that integrating
reasoning capabilities into reward modeling significantly enhances RM's
interpretability and performance. In this work, we introduce a new class of
generative reward models -- Reasoning Reward Models (ReasRMs) -- which
formulate reward modeling as a reasoning task. We propose a reasoning-oriented
training pipeline and train a family of ReasRMs, RM-R1. The training consists
of two key stages: (1) distillation of high-quality reasoning chains and (2)
reinforcement learning with verifiable rewards. RM-R1 improves LLM rollouts by
self-generating reasoning traces or chat-specific rubrics and evaluating
candidate responses against them. Empirically, our models achieve
state-of-the-art or near state-of-the-art performance of generative RMs across
multiple comprehensive reward model benchmarks, outperforming much larger
open-weight models (e.g., Llama3.1-405B) and proprietary ones (e.g., GPT-4o) by
up to 13.8%. Beyond final performance, we perform thorough empirical analysis
to understand the key ingredients of successful ReasRM training. To facilitate
future research, we release six ReasRM models along with code and data at
https://github.com/RM-R1-UIUC/RM-R1.
681988d6d6a5fee26b52ac28
Xiusi Chen
admin_assigned
2025-05-06T08:47:11.654Z
6270ff726417aed8a7340c8b
/avatars/3f14913c55cc4fc78678ac43fb603e80.svg
Xiusi Chen
user
XtremSup
681988d6d6a5fee26b52ac29
Gaotang Li
claimed_verified
2025-05-06T08:33:13.258Z
654d784d71a30c4bca09a319
/avatars/ab9f93122903ccd662267232bab30ad8.svg
Gaotang Li
user
gaotang
681988d6d6a5fee26b52ac2a
Ziqi Wang
681988d6d6a5fee26b52ac2b
Bowen Jin
681988d6d6a5fee26b52ac2c
Cheng Qian
681988d6d6a5fee26b52ac2d
Yu Wang
claimed_verified
2025-07-11T08:09:20.731Z
63234809155b0e2c44f354d6
/avatars/60d38f8f0e12363f3f5e0388e635d7b6.svg
Yu Wang
user
YuWangX
681988d6d6a5fee26b52ac2e
Hongru Wang
claimed_verified
2025-05-06T08:33:11.136Z
65f906e5c3dbdcae83ff7aac
https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
Hongru Wang
user
Merlin-Hongru
681988d6d6a5fee26b52ac2f
Yu Zhang
claimed_verified
2025-06-02T07:50:17.864Z
64b78f80b5d293ec95a3f694
https://cdn-avatars.huggingface.co/v1/production/uploads/64b78f80b5d293ec95a3f694/3rKg-m2i6ulj_aYhldvU8.png
Yu Zhang
user
yuz9yuz
681988d6d6a5fee26b52ac30
Denghui Zhang
admin_assigned
2025-05-06T08:48:00.793Z
66285acb73af5913c6bbf1ec
/avatars/8969e3a6ae2dcc0b1c49768fd044b9e0.svg
Denghui Zhang
user
zhangdenghui123
681988d6d6a5fee26b52ac31
Tong Zhang
681988d6d6a5fee26b52ac32
Hanghang Tong
681988d6d6a5fee26b52ac33
Heng Ji
RM-R1: Reward Modeling as Reasoning
Reward modeling is essential for aligning large language models (LLMs) with
human preferences, especially through reinforcement learning from human
feedback (RLHF). To provide accurate reward signals, a reward model (RM) should
stimulate deep thinking and conduct interpretable reasoning before assigning a
score or a judgment. However, existing RMs either produce opaque scalar scores
or directly generate the prediction of a preferred answer, making them struggle
to integrate natural language critiques, thus lacking interpretability.
Inspired by recent advances of long chain-of-thought (CoT) on
reasoning-intensive tasks, we hypothesize and validate that integrating
reasoning capabilities into reward modeling significantly enhances RM's
interpretability and performance. In this work, we introduce a new class of
generative reward models -- Reasoning Reward Models (ReasRMs) -- which
formulate reward modeling as a reasoning task. We propose a reasoning-oriented
training pipeline and train a family of ReasRMs, RM-R1. The training consists
of two key stages: (1) distillation of high-quality reasoning chains and (2)
reinforcement learning with verifiable rewards. RM-R1 improves LLM rollouts by
self-generating reasoning traces or chat-specific rubrics and evaluating
candidate responses against them. Empirically, our models achieve
state-of-the-art or near state-of-the-art performance of generative RMs across
multiple comprehensive reward model benchmarks, outperforming much larger
open-weight models (e.g., Llama3.1-405B) and proprietary ones (e.g., GPT-4o) by
up to 13.8%. Beyond final performance, we perform thorough empirical analysis
to understand the key ingredients of successful ReasRM training. To facilitate
future research, we release six ReasRM models along with code and data at
https://github.com/RM-R1-UIUC/RM-R1.
Reasoning Reward Models (ReasRMs) enhance reward modeling for large language models by integrating reasoning tasks, improving interpretability and performance.
reward modeling reinforcement learning from human feedback (RLHF) reward model (RM) long chain-of-thought (CoT) reasoning capabilities Reasoning Reward Models (ReasRMs) reasoning-oriented training pipeline high-quality reasoning chains reinforcement learning with verifiable rewards LLM rollouts reasoning traces chat-specific rubrics state-of-the-art performance reward model benchmarks Llama3.1-405B GPT-4o