HelpSteer2-Preference: Complementing Ratings with Preferences
Reward models are critical for aligning models to follow instructions, and
are typically trained following one of two popular paradigms: Bradley-Terry
style or Regression style. However, there is a lack of evidence that either
approach is better than the other, when adequately matched for data. This is
primarily because these approaches require data collected in different (but
incompatible) formats, meaning that adequately matched data is not available in
existing public datasets. To tackle this problem, we release preference
annotations (designed for Bradley-Terry training) to complement existing
ratings (designed for Regression style training) in the HelpSteer2 dataset. To
improve data interpretability, preference annotations are accompanied with
human-written justifications. Using this data, we conduct the first
head-to-head comparison of Bradley-Terry and Regression models when adequately
matched for data. Based on insights derived from such a comparison, we propose
a novel approach to combine Bradley-Terry and Regression reward modeling. A
Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on
RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We
also demonstrate the effectiveness of this reward model at aligning models to
follow instructions in RLHF. We open-source this dataset (CC-BY-4.0 license) at
https://huggingface.co/datasets/nvidia/HelpSteer2 and openly release the
trained Reward Model at
https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward
66fe0ad03e4ef90f0175b132
Zhilin Wang
claimed_verified
2024-10-03T15:27:08.582Z
635865c8aff68f72ac039a4e
/avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
Zhilin Wang
user
zhilinw
66fe0ad03e4ef90f0175b133
Alexander Bukharin
admin_assigned
2024-10-03T09:13:33.557Z
6501fe9d1671f7186dcfbc2d
/avatars/b31cf5f07ae77a8a1604e1472e59d4d8.svg
Alexander Bukharin
user
alexwb
66fe0ad03e4ef90f0175b134
Olivier Delalleau
admin_assigned
2024-10-03T09:13:39.596Z
6556379e10428134ff235afd
/avatars/ec569729870d7392e806e59a02f37d0c.svg
Olivier Delalleau
user
odelalleau
66fe0ad03e4ef90f0175b135
Daniel Egert
admin_assigned
2024-10-03T09:13:52.093Z
63d2f2bc2727d7888cb9f3cd
/avatars/f26eb927e803f1ea05d8ae9abfdd2846.svg
Daniel Egert
user
trias702
66fe0ad03e4ef90f0175b136
Gerald Shen
admin_assigned
2024-10-03T09:14:13.742Z
6635187e27cf4d1a75f27a40
/avatars/e36f11c962f52e783b4677ca5460c41e.svg
Gerald Shen
user
gshennvm
66fe0ad03e4ef90f0175b137
Jiaqi Zeng
66fe0ad03e4ef90f0175b138
Oleksii Kuchaiev
admin_assigned
2024-10-03T09:14:28.153Z
622937a4acd5bef90e55c49d
https://cdn-avatars.huggingface.co/v1/production/uploads/1649475932211-622937a4acd5bef90e55c49d.jpeg
Oleksii Kuchaiev
user
okuchaiev
66fe0ad03e4ef90f0175b139
Yi Dong
HelpSteer2-Preference: Complementing Ratings with Preferences
Reward models are critical for aligning models to follow instructions, and
are typically trained following one of two popular paradigms: Bradley-Terry
style or Regression style. However, there is a lack of evidence that either
approach is better than the other, when adequately matched for data. This is
primarily because these approaches require data collected in different (but
incompatible) formats, meaning that adequately matched data is not available in
existing public datasets. To tackle this problem, we release preference
annotations (designed for Bradley-Terry training) to complement existing
ratings (designed for Regression style training) in the HelpSteer2 dataset. To
improve data interpretability, preference annotations are accompanied with
human-written justifications. Using this data, we conduct the first
head-to-head comparison of Bradley-Terry and Regression models when adequately
matched for data. Based on insights derived from such a comparison, we propose
a novel approach to combine Bradley-Terry and Regression reward modeling. A
Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on
RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We
also demonstrate the effectiveness of this reward model at aligning models to
follow instructions in RLHF. We open-source this dataset (CC-BY-4.0 license) at
https://huggingface.co/datasets/nvidia/HelpSteer2 and openly release the
trained Reward Model at
https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward
A novel combined Bradley-Terry and Regression reward modeling approach outperforms other models on RewardBench and aligns models to follow instructions in RLHF.
Brunswick-Terry Regression preference annotations human-written justifications head-to-head comparison reward modeling RLHF RewardBench Llama-3.1-70B-Instruct NVIDIA/Llama-3.1-Nemotron-70B-Reward