title: Transforming and Combining Rewards for Aligning Large Language Models
summary: A common approach for aligning language models to human preferences is to
  first learn a reward model from preference data, and then use this reward model
  to update the language model. We study two closely related problems that arise
  in this approach. First, any monotone transformation of the reward model
  preserves preference ranking; is there a choice that is ``better'' than others?
  Second, we often wish to align language models to multiple properties: how
  should we combine multiple reward models? Using a probabilistic interpretation
  of the alignment procedure, we identify a natural choice for transformation for
  (the common case of) rewards learned from Bradley-Terry preference models. This
  derived transformation has two important properties. First, it emphasizes
  improving poorly-performing outputs, rather than outputs that already score
  well. This mitigates both underfitting (where some prompts are not improved)
  and reward hacking (where the model learns to exploit misspecification of the
  reward model). Second, it enables principled aggregation of rewards by linking
  summation to logical conjunction: the sum of transformed rewards corresponds to
  the probability that the output is ``good'' in all measured properties, in a
  sense we make precise. Experiments aligning language models to be both helpful
  and harmless using RLHF show substantial improvements over the baseline
  (non-transformed) approach.
paper_authors._id: 65bc673cd30db1c18d0431d9
paper_authors.name: Zihao Wang
paper_authors._id: 65bc673cd30db1c18d0431da
paper_authors.name: Chirag Nagpal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T13:22:36.325Z
paper_authors._id: 659709c567b8fc627910e4f2
paper_authors.avatarUrl: /avatars/5b44ae7941c565b83ac3125c2b7bd571.svg
paper_authors.fullname: Chirag Nagpal
paper_authors.type: user
paper_authors.user: chiragnagpal
paper_authors._id: 65bc673cd30db1c18d0431db
paper_authors.name: Jonathan Berant
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T13:22:44.208Z
paper_authors._id: 60811c9a4ad99100d63ce36c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619119757620-60811c9a4ad99100d63ce36c.png
paper_authors.fullname: Jonathan Berant
paper_authors.type: user
paper_authors.user: joberant
paper_authors._id: 65bc673cd30db1c18d0431dc
paper_authors.name: Jacob Eisenstein
paper_authors._id: 65bc673cd30db1c18d0431dd
paper_authors.name: Alex D'Amour
paper_authors._id: 65bc673cd30db1c18d0431de
paper_authors.name: Sanmi Koyejo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T13:23:15.115Z
paper_authors._id: 64931e7e2da595588288f161
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
paper_authors.fullname: Sanmi Koyejo
paper_authors.type: user
paper_authors.user: sanmikoyejo
paper_authors._id: 65bc673cd30db1c18d0431df
paper_authors.name: Victor Veitch
paper_title: Transforming and Combining Rewards for Aligning Large Language Models
paper_summary: A common approach for aligning language models to human preferences is to
  first learn a reward model from preference data, and then use this reward model
  to update the language model. We study two closely related problems that arise
  in this approach. First, any monotone transformation of the reward model
  preserves preference ranking; is there a choice that is ``better'' than others?
  Second, we often wish to align language models to multiple properties: how
  should we combine multiple reward models? Using a probabilistic interpretation
  of the alignment procedure, we identify a natural choice for transformation for
  (the common case of) rewards learned from Bradley-Terry preference models. This
  derived transformation has two important properties. First, it emphasizes
  improving poorly-performing outputs, rather than outputs that already score
  well. This mitigates both underfitting (where some prompts are not improved)
  and reward hacking (where the model learns to exploit misspecification of the
  reward model). Second, it enables principled aggregation of rewards by linking
  summation to logical conjunction: the sum of transformed rewards corresponds to
  the probability that the output is ``good'' in all measured properties, in a
  sense we make precise. Experiments aligning language models to be both helpful
  and harmless using RLHF show substantial improvements over the baseline
  (non-transformed) approach.
paper_ai_keywords: preference data reward model language model monotone transformation preference ranking vector dimension reduction alignment properties representation engineering (RepE) Bradley-Terry preference models output improvement underfitting reward robust maximizepecification logical conjunction summation RLHF helpful harmless baseline approach