title: Secrets of RLHF in Large Language Models Part II: Reward Modeling
summary: Reinforcement Learning from Human Feedback (RLHF) has become a crucial
  technology for aligning language models with human values and intentions,
  enabling models to produce more helpful and harmless responses. Reward models
  are trained as proxies for human preferences to drive reinforcement learning
  optimization. While reward models are often considered central to achieving
  high performance, they face the following challenges in practical applications:
  (1) Incorrect and ambiguous preference pairs in the dataset may hinder the
  reward model from accurately capturing human intent. (2) Reward models trained
  on data from a specific distribution often struggle to generalize to examples
  outside that distribution and are not suitable for iterative RLHF training.
    In this report, we attempt to address these two issues. (1) From a data
  perspective, we propose a method to measure the strength of preferences within
  the data, based on a voting mechanism of multiple reward models. Experimental
  results confirm that data with varying preference strengths have different
  impacts on reward model performance. We introduce a series of novel methods to
  mitigate the influence of incorrect and ambiguous preferences in the dataset
  and fully leverage high-quality preference data. (2) From an algorithmic
  standpoint, we introduce contrastive learning to enhance the ability of reward
  models to distinguish between chosen and rejected responses, thereby improving
  model generalization. Furthermore, we employ meta-learning to enable the reward
  model to maintain the ability to differentiate subtle differences in
  out-of-distribution samples, and this approach can be utilized for iterative
  RLHF optimization.
paper_authors._id: 65a0a832ff65c2d46da034b6
paper_authors.name: Binghai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T13:41:36.172Z
paper_authors._id: 642127bbe40f66bcd1e7e047
paper_authors.avatarUrl: /avatars/8da3b0ae9723af8d96906c7307e90bb8.svg
paper_authors.fullname: wang binghai
paper_authors.type: user
paper_authors.user: refrain-wbh
paper_authors._id: 65a0a832ff65c2d46da034b7
paper_authors.name: Rui Zheng
paper_authors._id: 65a0a832ff65c2d46da034b8
paper_authors.name: Lu Chen
paper_authors._id: 65a0a832ff65c2d46da034b9
paper_authors.name: Yan Liu
paper_authors._id: 65a0a832ff65c2d46da034ba
paper_authors.name: Shihan Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:23:45.513Z
paper_authors._id: 64ae0b6ccf90fe27556cf56e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
paper_authors.fullname: Shihan Dou
paper_authors.type: user
paper_authors.user: Ablustrund
paper_authors._id: 65a0a832ff65c2d46da034bb
paper_authors.name: Caishuang Huang
paper_authors._id: 65a0a832ff65c2d46da034bc
paper_authors.name: Wei Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:03:30.788Z
paper_authors._id: 62308ae8be7833fa98620372
paper_authors.avatarUrl: /avatars/220ed57376202c61ec1afbf5a4f45b67.svg
paper_authors.fullname: random
paper_authors.type: user
paper_authors.user: fakerbaby
paper_authors._id: 65a0a832ff65c2d46da034bd
paper_authors.name: Senjie Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:24:17.457Z
paper_authors._id: 63bbbe1fb8c61b8aa4922213
paper_authors.avatarUrl: /avatars/8199d6181794b8feee8fe9df73267af2.svg
paper_authors.fullname: jinsenjie
paper_authors.type: user
paper_authors.user: yizhidalaohu
paper_authors._id: 65a0a832ff65c2d46da034be
paper_authors.name: Enyu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:24:30.372Z
paper_authors._id: 64c28861c0126fdf8a285c92
paper_authors.avatarUrl: /avatars/409c914c827282f4276d3500561fabf3.svg
paper_authors.fullname: enyu
paper_authors.type: user
paper_authors.user: zhouenyu04
paper_authors._id: 65a0a832ff65c2d46da034bf
paper_authors.name: Chenyu Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:24:38.589Z
paper_authors._id: 65380e82cfc110ff796eaacc
paper_authors.avatarUrl: /avatars/9a39ed04ca60da503a4660a942ebb798.svg
paper_authors.fullname: Chenyu Shi
paper_authors.type: user
paper_authors.user: Chenyu2
paper_authors._id: 65a0a832ff65c2d46da034c0
paper_authors.name: Songyang Gao
paper_authors._id: 65a0a832ff65c2d46da034c1
paper_authors.name: Nuo Xu
paper_authors._id: 65a0a832ff65c2d46da034c2
paper_authors.name: Yuhao Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:25:28.842Z
paper_authors._id: 646dbfcaf813cfe153ef2964
paper_authors.avatarUrl: /avatars/8e73f91b35ac0d11671fd7399a43e723.svg
paper_authors.fullname: Yuhao Zhou
paper_authors.type: user
paper_authors.user: YuhaoZhou
paper_authors._id: 65a0a832ff65c2d46da034c3
paper_authors.name: Xiaoran Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:25:36.148Z
paper_authors._id: 6424fa5f988b8b19edba8ed4
paper_authors.avatarUrl: /avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
paper_authors.fullname: Xiaoran Fan
paper_authors.type: user
paper_authors.user: cnxup
paper_authors._id: 65a0a832ff65c2d46da034c4
paper_authors.name: Zhiheng Xi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:25:42.948Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 65a0a832ff65c2d46da034c5
paper_authors.name: Jun Zhao
paper_authors._id: 65a0a832ff65c2d46da034c6
paper_authors.name: Xiao Wang
paper_authors._id: 65a0a832ff65c2d46da034c7
paper_authors.name: Tao Ji
paper_authors._id: 65a0a832ff65c2d46da034c8
paper_authors.name: Hang Yan
paper_authors._id: 65a0a832ff65c2d46da034c9
paper_authors.name: Lixing Shen
paper_authors._id: 65a0a832ff65c2d46da034ca
paper_authors.name: Zhan Chen
paper_authors._id: 65a0a832ff65c2d46da034cb
paper_authors.name: Tao Gui
paper_authors._id: 65a0a832ff65c2d46da034cc
paper_authors.name: Qi Zhang
paper_authors._id: 65a0a832ff65c2d46da034cd
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:26:22.800Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_authors._id: 65a0a832ff65c2d46da034ce
paper_authors.name: Xuanjing Huang
paper_authors._id: 65a0a832ff65c2d46da034cf
paper_authors.name: Zuxuan Wu
paper_authors._id: 65a0a832ff65c2d46da034d0
paper_authors.name: Yu-Gang Jiang
paper_title: Secrets of RLHF in Large Language Models Part II: Reward Modeling
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has become a crucial
  technology for aligning language models with human values and intentions,
  enabling models to produce more helpful and harmless responses. Reward models
  are trained as proxies for human preferences to drive reinforcement learning
  optimization. While reward models are often considered central to achieving
  high performance, they face the following challenges in practical applications:
  (1) Incorrect and ambiguous preference pairs in the dataset may hinder the
  reward model from accurately capturing human intent. (2) Reward models trained
  on data from a specific distribution often struggle to generalize to examples
  outside that distribution and are not suitable for iterative RLHF training.
    In this report, we attempt to address these two issues. (1) From a data
  perspective, we propose a method to measure the strength of preferences within
  the data, based on a voting mechanism of multiple reward models. Experimental
  results confirm that data with varying preference strengths have different
  impacts on reward model performance. We introduce a series of novel methods to
  mitigate the influence of incorrect and ambiguous preferences in the dataset
  and fully leverage high-quality preference data. (2) From an algorithmic
  standpoint, we introduce contrastive learning to enhance the ability of reward
  models to distinguish between chosen and rejected responses, thereby improving
  model generalization. Furthermore, we employ meta-learning to enable the reward
  model to maintain the ability to differentiate subtle differences in
  out-of-distribution samples, and this approach can be utilized for iterative
  RLHF optimization.
paper_ai_summary: Methods are introduced to enhance reward models in Reinforcement Learning from Human Feedback by addressing data quality and generalization issues using voting mechanisms, contrastive learning, and meta-learning.
paper_ai_keywords: Reinforcement Learning from Human Feedback RLHF reward models preference pairs iterative RLHF training contrastive learning meta-learning voting mechanism out-of-distribution samples