title: HelpSteer3-Preference: Open Human-Annotated Preference Data across
    Diverse Tasks and Languages
summary: Preference datasets are essential for training general-domain,
  instruction-following language models with Reinforcement Learning from Human
  Feedback (RLHF). Each subsequent data release raises expectations for future
  data collection, meaning there is a constant need to advance the quality and
  diversity of openly available preference data. To address this need, we
  introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0),
  high-quality, human-annotated preference dataset comprising of over 40,000
  samples. These samples span diverse real-world applications of large language
  models (LLMs), including tasks relating to STEM, coding and multilingual
  scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that
  achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This
  represents a substantial improvement (~10% absolute) over the previously
  best-reported results from existing RMs. We demonstrate HelpSteer3-Preference
  can also be applied to train Generative RMs and how policy models can be
  aligned with RLHF using our RMs. Dataset (CC-BY-4.0):
  https://huggingface.co/datasets/nvidia/HelpSteer3#preference
paper_authors._id: 682b3c00466e84c3175b6d88
paper_authors.name: Zhilin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:29:40.457Z
paper_authors._id: 635865c8aff68f72ac039a4e
paper_authors.avatarUrl: /avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
paper_authors.fullname: Zhilin Wang
paper_authors.type: user
paper_authors.user: zhilinw
paper_authors._id: 682b3c00466e84c3175b6d89
paper_authors.name: Jiaqi Zeng
paper_authors._id: 682b3c00466e84c3175b6d8a
paper_authors.name: Olivier Delalleau
paper_authors._id: 682b3c00466e84c3175b6d8b
paper_authors.name: Hoo-Chang Shin
paper_authors._id: 682b3c00466e84c3175b6d8c
paper_authors.name: Felipe Soares
paper_authors._id: 682b3c00466e84c3175b6d8d
paper_authors.name: Alexander Bukharin
paper_authors._id: 682b3c00466e84c3175b6d8e
paper_authors.name: Ellie Evans
paper_authors._id: 682b3c00466e84c3175b6d8f
paper_authors.name: Yi Dong
paper_authors._id: 682b3c00466e84c3175b6d90
paper_authors.name: Oleksii Kuchaiev
paper_title: HelpSteer3-Preference: Open Human-Annotated Preference Data across
    Diverse Tasks and Languages
paper_summary: Preference datasets are essential for training general-domain,
  instruction-following language models with Reinforcement Learning from Human
  Feedback (RLHF). Each subsequent data release raises expectations for future
  data collection, meaning there is a constant need to advance the quality and
  diversity of openly available preference data. To address this need, we
  introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0),
  high-quality, human-annotated preference dataset comprising of over 40,000
  samples. These samples span diverse real-world applications of large language
  models (LLMs), including tasks relating to STEM, coding and multilingual
  scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that
  achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This
  represents a substantial improvement (~10% absolute) over the previously
  best-reported results from existing RMs. We demonstrate HelpSteer3-Preference
  can also be applied to train Generative RMs and how policy models can be
  aligned with RLHF using our RMs. Dataset (CC-BY-4.0):
  https://huggingface.co/datasets/nvidia/HelpSteer3#preference
paper_ai_summary: HelpSteer3-Preference, a high-quality human-annotated dataset, enhances Reward Models for Reinforcement Learning from Human Feedback, achieving top performance on RM-Bench and JudgeBench.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Reward Models (RMs) RM-Bench JudgeBench Generative RMs preference datasets large language models (LLMs)