HelpSteer3-Preference: Open Human-Annotated Preference Data across
  Diverse Tasks and Languages
Preference datasets are essential for training general-domain,
instruction-following language models with Reinforcement Learning from Human
Feedback (RLHF). Each subsequent data release raises expectations for future
data collection, meaning there is a constant need to advance the quality and
diversity of openly available preference data. To address this need, we
introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0),
high-quality, human-annotated preference dataset comprising of over 40,000
samples. These samples span diverse real-world applications of large language
models (LLMs), including tasks relating to STEM, coding and multilingual
scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that
achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This
represents a substantial improvement (~10% absolute) over the previously
best-reported results from existing RMs. We demonstrate HelpSteer3-Preference
can also be applied to train Generative RMs and how policy models can be
aligned with RLHF using our RMs. Dataset (CC-BY-4.0):
https://huggingface.co/datasets/nvidia/HelpSteer3#preference
682b3c00466e84c3175b6d88
Zhilin Wang
claimed_verified
2025-05-20T19:29:40.457Z
635865c8aff68f72ac039a4e
/avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
Zhilin Wang
user
zhilinw
682b3c00466e84c3175b6d89
Jiaqi Zeng
682b3c00466e84c3175b6d8a
Olivier Delalleau
682b3c00466e84c3175b6d8b
Hoo-Chang Shin
682b3c00466e84c3175b6d8c
Felipe Soares
682b3c00466e84c3175b6d8d
Alexander Bukharin
682b3c00466e84c3175b6d8e
Ellie Evans
682b3c00466e84c3175b6d8f
Yi Dong
682b3c00466e84c3175b6d90
Oleksii Kuchaiev
HelpSteer3-Preference: Open Human-Annotated Preference Data across
  Diverse Tasks and Languages
Preference datasets are essential for training general-domain,
instruction-following language models with Reinforcement Learning from Human
Feedback (RLHF). Each subsequent data release raises expectations for future
data collection, meaning there is a constant need to advance the quality and
diversity of openly available preference data. To address this need, we
introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0),
high-quality, human-annotated preference dataset comprising of over 40,000
samples. These samples span diverse real-world applications of large language
models (LLMs), including tasks relating to STEM, coding and multilingual
scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that
achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This
represents a substantial improvement (~10% absolute) over the previously
best-reported results from existing RMs. We demonstrate HelpSteer3-Preference
can also be applied to train Generative RMs and how policy models can be
aligned with RLHF using our RMs. Dataset (CC-BY-4.0):
https://huggingface.co/datasets/nvidia/HelpSteer3#preference
HelpSteer3-Preference, a high-quality human-annotated dataset, enhances Reward Models for Reinforcement Learning from Human Feedback, achieving top performance on RM-Bench and JudgeBench.
Reinforcement Learning from Human Feedback (RLHF) Reward Models (RMs) RM-Bench JudgeBench Generative RMs preference datasets large language models (LLMs)