title: RewardBench: Evaluating Reward Models for Language Modeling
summary: Reward models (RMs) are at the crux of successful RLHF to align pretrained
  models to human preferences, yet there has been relatively little study that
  focuses on evaluation of those reward models. Evaluating reward models presents
  an opportunity to understand the opaque technologies used for alignment of
  language models and which values are embedded in them. To date, very few
  descriptors of capabilities, training methods, or open-source reward models
  exist. In this paper, we present RewardBench, a benchmark dataset and code-base
  for evaluation, to enhance scientific understanding of reward models. The
  RewardBench dataset is a collection of prompt-win-lose trios spanning chat,
  reasoning, and safety, to benchmark how reward models perform on challenging,
  structured and out-of-distribution queries. We created specific comparison
  datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect
  facts) why one answer should be preferred to another. On the RewardBench
  leaderboard, we evaluate reward models trained with a variety of methods, such
  as the direct MLE training of classifiers and the implicit reward modeling of
  Direct Preference Optimization (DPO), and on a spectrum of datasets. We present
  many findings on propensity for refusals, reasoning limitations, and
  instruction following shortcomings of various reward models towards a better
  understanding of the RLHF process.
paper_authors._id: 65fb9e41bd57c920d77e66db
paper_authors.name: Nathan Lambert
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-21T15:13:29.906Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 65fb9e41bd57c920d77e66dc
paper_authors.name: Valentina Pyatkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:28:00.530Z
paper_authors._id: 6556cff80e7a7067a934445f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
paper_authors.fullname: Valentina Pyatkin
paper_authors.type: user
paper_authors.user: valpy
paper_authors._id: 65fb9e41bd57c920d77e66dd
paper_authors.name: Jacob Morrison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:28:15.851Z
paper_authors._id: 63f51f6cb51da4d61da72937
paper_authors.avatarUrl: /avatars/6023ecaafb28b7599f59816381d58bd0.svg
paper_authors.fullname: Jacob Morrison
paper_authors.type: user
paper_authors.user: jacobmorrison
paper_authors._id: 65fb9e41bd57c920d77e66de
paper_authors.name: LJ Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:20:34.173Z
paper_authors._id: 634e20a0c1ce28f1de920cc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
paper_authors.fullname: Lj V. Miranda
paper_authors.type: user
paper_authors.user: ljvmiranda921
paper_authors._id: 65fb9e41bd57c920d77e66df
paper_authors.name: Bill Yuchen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:28:27.962Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 65fb9e41bd57c920d77e66e0
paper_authors.name: Khyathi Chandu
paper_authors._id: 65fb9e41bd57c920d77e66e1
paper_authors.name: Nouha Dziri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:28:54.495Z
paper_authors._id: 625888c2c964680d85d70bff
paper_authors.avatarUrl: /avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
paper_authors.fullname: Nouha Dziri
paper_authors.type: user
paper_authors.user: nouhadziri
paper_authors._id: 65fb9e41bd57c920d77e66e2
paper_authors.name: Sachin Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:24:14.665Z
paper_authors._id: 64f2520f35653e77f02f288c
paper_authors.avatarUrl: /avatars/33620ae12f604e63eb9ec5ee9d89a820.svg
paper_authors.fullname: Sachin Kumar
paper_authors.type: user
paper_authors.user: shocheen
paper_authors._id: 65fb9e41bd57c920d77e66e3
paper_authors.name: Tom Zick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:29:28.977Z
paper_authors._id: 657b46ef30160611c6937e87
paper_authors.avatarUrl: /avatars/c85264f3008d15582e39661888d93bd7.svg
paper_authors.fullname: Tom Zick
paper_authors.type: user
paper_authors.user: tzick
paper_authors._id: 65fb9e41bd57c920d77e66e4
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:29:34.936Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 65fb9e41bd57c920d77e66e5
paper_authors.name: Noah A. Smith
paper_authors._id: 65fb9e41bd57c920d77e66e6
paper_authors.name: Hannaneh Hajishirzi
paper_title: RewardBench: Evaluating Reward Models for Language Modeling
paper_summary: Reward models (RMs) are at the crux of successful RLHF to align pretrained
  models to human preferences, yet there has been relatively little study that
  focuses on evaluation of those reward models. Evaluating reward models presents
  an opportunity to understand the opaque technologies used for alignment of
  language models and which values are embedded in them. To date, very few
  descriptors of capabilities, training methods, or open-source reward models
  exist. In this paper, we present RewardBench, a benchmark dataset and code-base
  for evaluation, to enhance scientific understanding of reward models. The
  RewardBench dataset is a collection of prompt-win-lose trios spanning chat,
  reasoning, and safety, to benchmark how reward models perform on challenging,
  structured and out-of-distribution queries. We created specific comparison
  datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect
  facts) why one answer should be preferred to another. On the RewardBench
  leaderboard, we evaluate reward models trained with a variety of methods, such
  as the direct MLE training of classifiers and the implicit reward modeling of
  Direct Preference Optimization (DPO), and on a spectrum of datasets. We present
  many findings on propensity for refusals, reasoning limitations, and
  instruction following shortcomings of various reward models towards a better
  understanding of the RLHF process.
paper_ai_summary: A benchmark dataset and code-base, RewardBench, evaluates reward models to understand alignment with human preferences, revealing capabilities and limitations of different training methods.
paper_ai_keywords: reward models RLHF RewardBench prompt-win-lose trios chat reasoning safety direct MLE training Direct Preference Optimization refusals reasoning limitations instruction following