Secrets of RLHF in Large Language Models Part II: Reward Modeling
Reinforcement Learning from Human Feedback (RLHF) has become a crucial
technology for aligning language models with human values and intentions,
enabling models to produce more helpful and harmless responses. Reward models
are trained as proxies for human preferences to drive reinforcement learning
optimization. While reward models are often considered central to achieving
high performance, they face the following challenges in practical applications:
(1) Incorrect and ambiguous preference pairs in the dataset may hinder the
reward model from accurately capturing human intent. (2) Reward models trained
on data from a specific distribution often struggle to generalize to examples
outside that distribution and are not suitable for iterative RLHF training.
  In this report, we attempt to address these two issues. (1) From a data
perspective, we propose a method to measure the strength of preferences within
the data, based on a voting mechanism of multiple reward models. Experimental
results confirm that data with varying preference strengths have different
impacts on reward model performance. We introduce a series of novel methods to
mitigate the influence of incorrect and ambiguous preferences in the dataset
and fully leverage high-quality preference data. (2) From an algorithmic
standpoint, we introduce contrastive learning to enhance the ability of reward
models to distinguish between chosen and rejected responses, thereby improving
model generalization. Furthermore, we employ meta-learning to enable the reward
model to maintain the ability to differentiate subtle differences in
out-of-distribution samples, and this approach can be utilized for iterative
RLHF optimization.
65a0a832ff65c2d46da034b6
Binghai Wang
claimed_verified
2025-05-16T13:41:36.172Z
642127bbe40f66bcd1e7e047
/avatars/8da3b0ae9723af8d96906c7307e90bb8.svg
wang binghai
user
refrain-wbh
65a0a832ff65c2d46da034b7
Rui Zheng
65a0a832ff65c2d46da034b8
Lu Chen
65a0a832ff65c2d46da034b9
Yan Liu
65a0a832ff65c2d46da034ba
Shihan Dou
admin_assigned
2024-01-12T10:23:45.513Z
64ae0b6ccf90fe27556cf56e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
Shihan Dou
user
Ablustrund
65a0a832ff65c2d46da034bb
Caishuang Huang
65a0a832ff65c2d46da034bc
Wei Shen
claimed_verified
2024-06-28T08:03:30.788Z
62308ae8be7833fa98620372
/avatars/220ed57376202c61ec1afbf5a4f45b67.svg
random
user
fakerbaby
65a0a832ff65c2d46da034bd
Senjie Jin
admin_assigned
2024-01-12T10:24:17.457Z
63bbbe1fb8c61b8aa4922213
/avatars/8199d6181794b8feee8fe9df73267af2.svg
jinsenjie
user
yizhidalaohu
65a0a832ff65c2d46da034be
Enyu Zhou
admin_assigned
2024-01-12T10:24:30.372Z
64c28861c0126fdf8a285c92
/avatars/409c914c827282f4276d3500561fabf3.svg
enyu
user
zhouenyu04
65a0a832ff65c2d46da034bf
Chenyu Shi
admin_assigned
2024-01-12T10:24:38.589Z
65380e82cfc110ff796eaacc
/avatars/9a39ed04ca60da503a4660a942ebb798.svg
Chenyu Shi
user
Chenyu2
65a0a832ff65c2d46da034c0
Songyang Gao
65a0a832ff65c2d46da034c1
Nuo Xu
65a0a832ff65c2d46da034c2
Yuhao Zhou
admin_assigned
2024-01-12T10:25:28.842Z
646dbfcaf813cfe153ef2964
/avatars/8e73f91b35ac0d11671fd7399a43e723.svg
Yuhao Zhou
user
YuhaoZhou
65a0a832ff65c2d46da034c3
Xiaoran Fan
admin_assigned
2024-01-12T10:25:36.148Z
6424fa5f988b8b19edba8ed4
/avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
Xiaoran Fan
user
cnxup
65a0a832ff65c2d46da034c4
Zhiheng Xi
admin_assigned
2024-01-12T10:25:42.948Z
653a6e5cae155b92bae77b74
https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
Zhiheng Xi
user
WooooDyy
65a0a832ff65c2d46da034c5
Jun Zhao
65a0a832ff65c2d46da034c6
Xiao Wang
65a0a832ff65c2d46da034c7
Tao Ji
65a0a832ff65c2d46da034c8
Hang Yan
65a0a832ff65c2d46da034c9
Lixing Shen
65a0a832ff65c2d46da034ca
Zhan Chen
65a0a832ff65c2d46da034cb
Tao Gui
65a0a832ff65c2d46da034cc
Qi Zhang
65a0a832ff65c2d46da034cd
Xipeng Qiu
admin_assigned
2024-01-12T10:26:22.800Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
65a0a832ff65c2d46da034ce
Xuanjing Huang
65a0a832ff65c2d46da034cf
Zuxuan Wu
65a0a832ff65c2d46da034d0
Yu-Gang Jiang
Secrets of RLHF in Large Language Models Part II: Reward Modeling
Reinforcement Learning from Human Feedback (RLHF) has become a crucial
technology for aligning language models with human values and intentions,
enabling models to produce more helpful and harmless responses. Reward models
are trained as proxies for human preferences to drive reinforcement learning
optimization. While reward models are often considered central to achieving
high performance, they face the following challenges in practical applications:
(1) Incorrect and ambiguous preference pairs in the dataset may hinder the
reward model from accurately capturing human intent. (2) Reward models trained
on data from a specific distribution often struggle to generalize to examples
outside that distribution and are not suitable for iterative RLHF training.
  In this report, we attempt to address these two issues. (1) From a data
perspective, we propose a method to measure the strength of preferences within
the data, based on a voting mechanism of multiple reward models. Experimental
results confirm that data with varying preference strengths have different
impacts on reward model performance. We introduce a series of novel methods to
mitigate the influence of incorrect and ambiguous preferences in the dataset
and fully leverage high-quality preference data. (2) From an algorithmic
standpoint, we introduce contrastive learning to enhance the ability of reward
models to distinguish between chosen and rejected responses, thereby improving
model generalization. Furthermore, we employ meta-learning to enable the reward
model to maintain the ability to differentiate subtle differences in
out-of-distribution samples, and this approach can be utilized for iterative
RLHF optimization.
Methods are introduced to enhance reward models in Reinforcement Learning from Human Feedback by addressing data quality and generalization issues using voting mechanisms, contrastive learning, and meta-learning.
Reinforcement Learning from Human Feedback RLHF reward models preference pairs iterative RLHF training contrastive learning meta-learning voting mechanism out-of-distribution samples