title: HelpSteer2-Preference: Complementing Ratings with Preferences
summary: Reward models are critical for aligning models to follow instructions, and
  are typically trained following one of two popular paradigms: Bradley-Terry
  style or Regression style. However, there is a lack of evidence that either
  approach is better than the other, when adequately matched for data. This is
  primarily because these approaches require data collected in different (but
  incompatible) formats, meaning that adequately matched data is not available in
  existing public datasets. To tackle this problem, we release preference
  annotations (designed for Bradley-Terry training) to complement existing
  ratings (designed for Regression style training) in the HelpSteer2 dataset. To
  improve data interpretability, preference annotations are accompanied with
  human-written justifications. Using this data, we conduct the first
  head-to-head comparison of Bradley-Terry and Regression models when adequately
  matched for data. Based on insights derived from such a comparison, we propose
  a novel approach to combine Bradley-Terry and Regression reward modeling. A
  Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on
  RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We
  also demonstrate the effectiveness of this reward model at aligning models to
  follow instructions in RLHF. We open-source this dataset (CC-BY-4.0 license) at
  https://huggingface.co/datasets/nvidia/HelpSteer2 and openly release the
  trained Reward Model at
  https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward
paper_authors._id: 66fe0ad03e4ef90f0175b132
paper_authors.name: Zhilin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T15:27:08.582Z
paper_authors._id: 635865c8aff68f72ac039a4e
paper_authors.avatarUrl: /avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
paper_authors.fullname: Zhilin Wang
paper_authors.type: user
paper_authors.user: zhilinw
paper_authors._id: 66fe0ad03e4ef90f0175b133
paper_authors.name: Alexander Bukharin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:13:33.557Z
paper_authors._id: 6501fe9d1671f7186dcfbc2d
paper_authors.avatarUrl: /avatars/b31cf5f07ae77a8a1604e1472e59d4d8.svg
paper_authors.fullname: Alexander Bukharin
paper_authors.type: user
paper_authors.user: alexwb
paper_authors._id: 66fe0ad03e4ef90f0175b134
paper_authors.name: Olivier Delalleau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:13:39.596Z
paper_authors._id: 6556379e10428134ff235afd
paper_authors.avatarUrl: /avatars/ec569729870d7392e806e59a02f37d0c.svg
paper_authors.fullname: Olivier Delalleau
paper_authors.type: user
paper_authors.user: odelalleau
paper_authors._id: 66fe0ad03e4ef90f0175b135
paper_authors.name: Daniel Egert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:13:52.093Z
paper_authors._id: 63d2f2bc2727d7888cb9f3cd
paper_authors.avatarUrl: /avatars/f26eb927e803f1ea05d8ae9abfdd2846.svg
paper_authors.fullname: Daniel Egert
paper_authors.type: user
paper_authors.user: trias702
paper_authors._id: 66fe0ad03e4ef90f0175b136
paper_authors.name: Gerald Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:14:13.742Z
paper_authors._id: 6635187e27cf4d1a75f27a40
paper_authors.avatarUrl: /avatars/e36f11c962f52e783b4677ca5460c41e.svg
paper_authors.fullname: Gerald Shen
paper_authors.type: user
paper_authors.user: gshennvm
paper_authors._id: 66fe0ad03e4ef90f0175b137
paper_authors.name: Jiaqi Zeng
paper_authors._id: 66fe0ad03e4ef90f0175b138
paper_authors.name: Oleksii Kuchaiev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:14:28.153Z
paper_authors._id: 622937a4acd5bef90e55c49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649475932211-622937a4acd5bef90e55c49d.jpeg
paper_authors.fullname: Oleksii Kuchaiev
paper_authors.type: user
paper_authors.user: okuchaiev
paper_authors._id: 66fe0ad03e4ef90f0175b139
paper_authors.name: Yi Dong
paper_title: HelpSteer2-Preference: Complementing Ratings with Preferences
paper_summary: Reward models are critical for aligning models to follow instructions, and
  are typically trained following one of two popular paradigms: Bradley-Terry
  style or Regression style. However, there is a lack of evidence that either
  approach is better than the other, when adequately matched for data. This is
  primarily because these approaches require data collected in different (but
  incompatible) formats, meaning that adequately matched data is not available in
  existing public datasets. To tackle this problem, we release preference
  annotations (designed for Bradley-Terry training) to complement existing
  ratings (designed for Regression style training) in the HelpSteer2 dataset. To
  improve data interpretability, preference annotations are accompanied with
  human-written justifications. Using this data, we conduct the first
  head-to-head comparison of Bradley-Terry and Regression models when adequately
  matched for data. Based on insights derived from such a comparison, we propose
  a novel approach to combine Bradley-Terry and Regression reward modeling. A
  Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on
  RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We
  also demonstrate the effectiveness of this reward model at aligning models to
  follow instructions in RLHF. We open-source this dataset (CC-BY-4.0 license) at
  https://huggingface.co/datasets/nvidia/HelpSteer2 and openly release the
  trained Reward Model at
  https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward
paper_ai_summary: A novel combined Bradley-Terry and Regression reward modeling approach outperforms other models on RewardBench and aligns models to follow instructions in RLHF.
paper_ai_keywords: Brunswick-Terry Regression preference annotations human-written justifications head-to-head comparison reward modeling RLHF RewardBench Llama-3.1-70B-Instruct NVIDIA/Llama-3.1-Nemotron-70B-Reward