title: HelpSteer2: Open-source dataset for training top-performing reward
    models
summary: High-quality preference datasets are essential for training reward models
  that can effectively guide large language models (LLMs) in generating
  high-quality responses aligned with human preferences. As LLMs become stronger
  and better aligned, permissively licensed preference datasets, such as Open
  Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for
  reward modeling. Methods that distil preference data from proprietary LLMs such
  as GPT-4 have restrictions on commercial usage imposed by model providers. To
  improve upon both generated responses and attribute labeling quality, we
  release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0).
  Using a powerful internal base model trained on HelpSteer2, we are able to
  achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming
  currently listed open and proprietary models, as of June 12th, 2024. Notably,
  HelpSteer2 consists of only ten thousand response pairs, an order of magnitude
  fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly
  efficient for training reward models. Our extensive experiments demonstrate
  that reward models trained with HelpSteer2 are effective in aligning LLMs. In
  particular, we propose SteerLM 2.0, a model alignment approach that can
  effectively make use of the rich multi-attribute score predicted by our reward
  models. HelpSteer2 is available at
  https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at
  https://github.com/NVIDIA/NeMo-Aligner
paper_authors._id: 666ba265fba6c458fcee3cf6
paper_authors.name: Zhilin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:09:11.245Z
paper_authors._id: 635865c8aff68f72ac039a4e
paper_authors.avatarUrl: /avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
paper_authors.fullname: Zhilin Wang
paper_authors.type: user
paper_authors.user: zhilinw
paper_authors._id: 666ba265fba6c458fcee3cf7
paper_authors.name: Yi Dong
paper_authors._id: 666ba265fba6c458fcee3cf8
paper_authors.name: Olivier Delalleau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:09:45.540Z
paper_authors._id: 6556379e10428134ff235afd
paper_authors.avatarUrl: /avatars/ec569729870d7392e806e59a02f37d0c.svg
paper_authors.fullname: Olivier Delalleau
paper_authors.type: user
paper_authors.user: odelalleau
paper_authors._id: 666ba265fba6c458fcee3cf9
paper_authors.name: Jiaqi Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T07:46:01.951Z
paper_authors._id: 6511cb8d254ae1f96174a330
paper_authors.avatarUrl: /avatars/525c88f6c146fa83e0954d8ed97ee99e.svg
paper_authors.fullname: jiaqiz
paper_authors.type: user
paper_authors.user: jiaqiz
paper_authors._id: 666ba265fba6c458fcee3cfa
paper_authors.name: Gerald Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:10:12.778Z
paper_authors._id: 6635187e27cf4d1a75f27a40
paper_authors.avatarUrl: /avatars/e36f11c962f52e783b4677ca5460c41e.svg
paper_authors.fullname: Gerald Shen
paper_authors.type: user
paper_authors.user: gshennvm
paper_authors._id: 666ba265fba6c458fcee3cfb
paper_authors.name: Daniel Egert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:10:24.243Z
paper_authors._id: 63d2f2bc2727d7888cb9f3cd
paper_authors.avatarUrl: /avatars/f26eb927e803f1ea05d8ae9abfdd2846.svg
paper_authors.fullname: Daniel Egert
paper_authors.type: user
paper_authors.user: trias702
paper_authors._id: 666ba265fba6c458fcee3cfc
paper_authors.name: Jimmy J. Zhang
paper_authors._id: 666ba265fba6c458fcee3cfd
paper_authors.name: Makesh Narsimhan Sreedhar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:11:05.118Z
paper_authors._id: 6643782e17bbed56d0b9932b
paper_authors.avatarUrl: /avatars/9f20fcf2b8bfc75c56c510e7bda4dbd7.svg
paper_authors.fullname: Makesh Sreedhar
paper_authors.type: user
paper_authors.user: makeshn
paper_authors._id: 666ba265fba6c458fcee3cfe
paper_authors.name: Oleksii Kuchaiev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:11:10.896Z
paper_authors._id: 622937a4acd5bef90e55c49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649475932211-622937a4acd5bef90e55c49d.jpeg
paper_authors.fullname: Oleksii Kuchaiev
paper_authors.type: user
paper_authors.user: okuchaiev
paper_title: HelpSteer2: Open-source dataset for training top-performing reward
    models
paper_summary: High-quality preference datasets are essential for training reward models
  that can effectively guide large language models (LLMs) in generating
  high-quality responses aligned with human preferences. As LLMs become stronger
  and better aligned, permissively licensed preference datasets, such as Open
  Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for
  reward modeling. Methods that distil preference data from proprietary LLMs such
  as GPT-4 have restrictions on commercial usage imposed by model providers. To
  improve upon both generated responses and attribute labeling quality, we
  release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0).
  Using a powerful internal base model trained on HelpSteer2, we are able to
  achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming
  currently listed open and proprietary models, as of June 12th, 2024. Notably,
  HelpSteer2 consists of only ten thousand response pairs, an order of magnitude
  fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly
  efficient for training reward models. Our extensive experiments demonstrate
  that reward models trained with HelpSteer2 are effective in aligning LLMs. In
  particular, we propose SteerLM 2.0, a model alignment approach that can
  effectively make use of the rich multi-attribute score predicted by our reward
  models. HelpSteer2 is available at
  https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at
  https://github.com/NVIDIA/NeMo-Aligner
paper_ai_summary: NVIDIA's HelpSteer2 dataset and SteerLM 2.0 model improve LLM response quality and alignment using permissively licensed preference data.
paper_ai_keywords: preference datasets reward models large language models (LLMs) Open Assistant HH-RLHF HelpSteer HuRLHF HelpSteer2 SOTA score Reward-Bench SteerLM model alignment multi-attribute score