Pre-Trained Policy Discriminators are General Reward Models
We offer a novel perspective on reward modeling by formulating it as a policy
discriminator, which quantifies the difference between two policies to generate
a reward signal, guiding the training policy towards a target policy with
desired behaviors. Based on this conceptual insight, we propose a scalable
pre-training method named Policy Discriminative Learning (POLAR), which trains
a reward model (RM) to discern identical policies and discriminate different
ones. Unlike traditional reward modeling methods relying on absolute
preferences, POLAR captures the relative difference between one policy and an
arbitrary target policy, which is a scalable, high-level optimization objective
suitable for modeling generic ranking relationships. Leveraging the POLAR
pre-training paradigm, we present a series of RMs with parameter scales from
1.8B to 7B. Empirical results show that POLAR substantially outperforms
traditional non-pre-trained methods, significantly enhancing RM performance.
For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on
STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA
baselines. POLAR also shows robust generalization capabilities in RLHF using
Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly
enhancing policy performance--improving LLaMa3.1-8B from an average of 47.36%
to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. Moreover,
scaling experiments reveal a clear power-law relationship between computation
and performance, supported by linear correlation coefficients approaching 0.99.
The impressive performance, strong generalization, and scaling properties
suggest that POLAR is a promising direction for developing general and strong
reward models.
686c7f78364e2ad167eb5354
Shihan Dou
claimed_verified
2025-07-09T15:10:18.506Z
64ae0b6ccf90fe27556cf56e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
Shihan Dou
user
Ablustrund
686c7f78364e2ad167eb5355
Shichun Liu
claimed_verified
2025-07-15T19:11:24.523Z
65435cad429b80b14922ab8d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/N8oWq4ZZn3dRxmXi18FrA.jpeg
Shichun Liu
user
Liusc2020
686c7f78364e2ad167eb5356
Yuming Yang
claimed_verified
2025-07-08T08:06:28.979Z
655c6b1abfb531437a54c0e6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/V8Md2mMX83hrSowKk6qMS.jpeg
Yuming Yang
user
Umean
686c7f78364e2ad167eb5357
Yicheng Zou
admin_assigned
2025-07-08T14:08:52.408Z
6234238485575ce6ff1f169a
/avatars/e5fff05f21cdea4e5aebc8ba426fac29.svg
Yicheng Zou
user
RowitZou
686c7f78364e2ad167eb5358
Yunhua Zhou
686c7f78364e2ad167eb5359
Shuhao Xing
686c7f78364e2ad167eb535a
Chenhao Huang
686c7f78364e2ad167eb535b
Qiming Ge
686c7f78364e2ad167eb535c
Demin Song
686c7f78364e2ad167eb535d
Haijun Lv
686c7f78364e2ad167eb535e
Songyang Gao
686c7f78364e2ad167eb535f
Chengqi Lv
686c7f78364e2ad167eb5360
Enyu Zhou
686c7f78364e2ad167eb5361
Honglin Guo
686c7f78364e2ad167eb5362
Zhiheng Xi
686c7f78364e2ad167eb5363
Wenwei Zhang
686c7f78364e2ad167eb5364
Qipeng Guo
686c7f78364e2ad167eb5365
Qi Zhang
686c7f78364e2ad167eb5366
Xipeng Qiu
686c7f78364e2ad167eb5367
Xuanjing Huang
admin_assigned
2025-07-08T14:06:00.646Z
67f9c4ee171948c38302ae0f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Cqb3ijr_sZkpLhEEEEybK.png
Xuanjing Huang
user
xjhuang
686c7f78364e2ad167eb5368
Tao Gui
686c7f78364e2ad167eb5369
Kai Chen
Pre-Trained Policy Discriminators are General Reward Models
We offer a novel perspective on reward modeling by formulating it as a policy
discriminator, which quantifies the difference between two policies to generate
a reward signal, guiding the training policy towards a target policy with
desired behaviors. Based on this conceptual insight, we propose a scalable
pre-training method named Policy Discriminative Learning (POLAR), which trains
a reward model (RM) to discern identical policies and discriminate different
ones. Unlike traditional reward modeling methods relying on absolute
preferences, POLAR captures the relative difference between one policy and an
arbitrary target policy, which is a scalable, high-level optimization objective
suitable for modeling generic ranking relationships. Leveraging the POLAR
pre-training paradigm, we present a series of RMs with parameter scales from
1.8B to 7B. Empirical results show that POLAR substantially outperforms
traditional non-pre-trained methods, significantly enhancing RM performance.
For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on
STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA
baselines. POLAR also shows robust generalization capabilities in RLHF using
Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly
enhancing policy performance--improving LLaMa3.1-8B from an average of 47.36%
to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. Moreover,
scaling experiments reveal a clear power-law relationship between computation
and performance, supported by linear correlation coefficients approaching 0.99.
The impressive performance, strong generalization, and scaling properties
suggest that POLAR is a promising direction for developing general and strong
reward models.
A novel reward modeling approach, Policy Discriminative Learning (POLAR), enhances reward model performance and generalization in reinforcement learning by focusing on relative policy differences.
policy discriminator reward model POLAR parameter scales preference accuracy RLHF Reinforcement Fine-tuning RFT power-law relationship linear correlation coefficients