title: Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate
    Reward Hacking
summary: Reward models play a key role in aligning language model applications towards
  human preferences. However, this setup creates an incentive for the language
  model to exploit errors in the reward model to achieve high estimated reward, a
  phenomenon often termed reward hacking. A natural mitigation is to train
  an ensemble of reward models, aggregating over model outputs to obtain a more
  robust reward estimate. We explore the application of reward ensembles to
  alignment at both training time (through reinforcement learning) and inference
  time (through reranking). First, we show that reward models are
  underspecified: reward models that perform similarly in-distribution can
  yield very different rewards when used in alignment, due to distribution shift.
  Second, underspecification results in overoptimization, where alignment to one
  reward model does not improve reward as measured by another reward model
  trained on the same data. Third, overoptimization is mitigated by the use of
  reward ensembles, and ensembles that vary by their pretraining seeds
  lead to better generalization than ensembles that differ only by their
  fine-tuning seeds, with both outperforming individual reward models.
  However, even pretrain reward ensembles do not eliminate reward hacking: we
  show several qualitative reward hacking phenomena that are not mitigated by
  ensembling because all reward models in the ensemble exhibit similar error
  patterns.
paper_authors._id: 657bcf8c538666d04ccc4723
paper_authors.name: Jacob Eisenstein
paper_authors._id: 657bcf8c538666d04ccc4724
paper_authors.name: Chirag Nagpal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:21:22.501Z
paper_authors._id: 65080f431704c7eb0aaa0720
paper_authors.avatarUrl: /avatars/b1f946b689fd9f99f9fff4df02bbc4f1.svg
paper_authors.fullname: Chirag Nagpal
paper_authors.type: user
paper_authors.user: chiragnagpal02
paper_authors._id: 657bcf8c538666d04ccc4725
paper_authors.name: Alekh Agarwal
paper_authors._id: 657bcf8c538666d04ccc4726
paper_authors.name: Ahmad Beirami
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:21:12.084Z
paper_authors._id: 65482d45407bb19ff5adb8a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_etKjaaubh3osxp3R2MP7.jpeg
paper_authors.fullname: Ahmad Beirami
paper_authors.type: user
paper_authors.user: beirami
paper_authors._id: 657bcf8c538666d04ccc4727
paper_authors.name: Alex D'Amour
paper_authors._id: 657bcf8c538666d04ccc4728
paper_authors.name: DJ Dvijotham
paper_authors._id: 657bcf8c538666d04ccc4729
paper_authors.name: Adam Fisch
paper_authors._id: 657bcf8c538666d04ccc472a
paper_authors.name: Katherine Heller
paper_authors._id: 657bcf8c538666d04ccc472b
paper_authors.name: Stephen Pfohl
paper_authors._id: 657bcf8c538666d04ccc472c
paper_authors.name: Deepak Ramachandran
paper_authors._id: 657bcf8c538666d04ccc472d
paper_authors.name: Peter Shaw
paper_authors._id: 657bcf8c538666d04ccc472e
paper_authors.name: Jonathan Berant
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:20:18.063Z
paper_authors._id: 60811c9a4ad99100d63ce36c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619119757620-60811c9a4ad99100d63ce36c.png
paper_authors.fullname: Jonathan Berant
paper_authors.type: user
paper_authors.user: joberant
paper_title: Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate
    Reward Hacking
paper_summary: Reward models play a key role in aligning language model applications towards
  human preferences. However, this setup creates an incentive for the language
  model to exploit errors in the reward model to achieve high estimated reward, a
  phenomenon often termed reward hacking. A natural mitigation is to train
  an ensemble of reward models, aggregating over model outputs to obtain a more
  robust reward estimate. We explore the application of reward ensembles to
  alignment at both training time (through reinforcement learning) and inference
  time (through reranking). First, we show that reward models are
  underspecified: reward models that perform similarly in-distribution can
  yield very different rewards when used in alignment, due to distribution shift.
  Second, underspecification results in overoptimization, where alignment to one
  reward model does not improve reward as measured by another reward model
  trained on the same data. Third, overoptimization is mitigated by the use of
  reward ensembles, and ensembles that vary by their pretraining seeds
  lead to better generalization than ensembles that differ only by their
  fine-tuning seeds, with both outperforming individual reward models.
  However, even pretrain reward ensembles do not eliminate reward hacking: we
  show several qualitative reward hacking phenomena that are not mitigated by
  ensembling because all reward models in the ensemble exhibit similar error
  patterns.
paper_ai_summary: Reward ensembles improve language model alignment with human preferences but do not eliminate reward hacking.
paper_ai_keywords: reward hacking reinforcement learning reranking underspecified overoptimization pretraining seeds fine-tuning seeds