Transforming and Combining Rewards for Aligning Large Language Models
A common approach for aligning language models to human preferences is to
first learn a reward model from preference data, and then use this reward model
to update the language model. We study two closely related problems that arise
in this approach. First, any monotone transformation of the reward model
preserves preference ranking; is there a choice that is ``better'' than others?
Second, we often wish to align language models to multiple properties: how
should we combine multiple reward models? Using a probabilistic interpretation
of the alignment procedure, we identify a natural choice for transformation for
(the common case of) rewards learned from Bradley-Terry preference models. This
derived transformation has two important properties. First, it emphasizes
improving poorly-performing outputs, rather than outputs that already score
well. This mitigates both underfitting (where some prompts are not improved)
and reward hacking (where the model learns to exploit misspecification of the
reward model). Second, it enables principled aggregation of rewards by linking
summation to logical conjunction: the sum of transformed rewards corresponds to
the probability that the output is ``good'' in all measured properties, in a
sense we make precise. Experiments aligning language models to be both helpful
and harmless using RLHF show substantial improvements over the baseline
(non-transformed) approach.
65bc673cd30db1c18d0431d9
Zihao Wang
65bc673cd30db1c18d0431da
Chirag Nagpal
admin_assigned
2024-02-02T13:22:36.325Z
659709c567b8fc627910e4f2
/avatars/5b44ae7941c565b83ac3125c2b7bd571.svg
Chirag Nagpal
user
chiragnagpal
65bc673cd30db1c18d0431db
Jonathan Berant
admin_assigned
2024-02-02T13:22:44.208Z
60811c9a4ad99100d63ce36c
https://cdn-avatars.huggingface.co/v1/production/uploads/1619119757620-60811c9a4ad99100d63ce36c.png
Jonathan Berant
user
joberant
65bc673cd30db1c18d0431dc
Jacob Eisenstein
65bc673cd30db1c18d0431dd
Alex D'Amour
65bc673cd30db1c18d0431de
Sanmi Koyejo
admin_assigned
2024-02-02T13:23:15.115Z
64931e7e2da595588288f161
https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
Sanmi Koyejo
user
sanmikoyejo
65bc673cd30db1c18d0431df
Victor Veitch
Transforming and Combining Rewards for Aligning Large Language Models
A common approach for aligning language models to human preferences is to
first learn a reward model from preference data, and then use this reward model
to update the language model. We study two closely related problems that arise
in this approach. First, any monotone transformation of the reward model
preserves preference ranking; is there a choice that is ``better'' than others?
Second, we often wish to align language models to multiple properties: how
should we combine multiple reward models? Using a probabilistic interpretation
of the alignment procedure, we identify a natural choice for transformation for
(the common case of) rewards learned from Bradley-Terry preference models. This
derived transformation has two important properties. First, it emphasizes
improving poorly-performing outputs, rather than outputs that already score
well. This mitigates both underfitting (where some prompts are not improved)
and reward hacking (where the model learns to exploit misspecification of the
reward model). Second, it enables principled aggregation of rewards by linking
summation to logical conjunction: the sum of transformed rewards corresponds to
the probability that the output is ``good'' in all measured properties, in a
sense we make precise. Experiments aligning language models to be both helpful
and harmless using RLHF show substantial improvements over the baseline
(non-transformed) approach.
The study examines the transformation and aggregation of reward models to align language models with human preferences, identifying a method that improves poor outputs and aggregates rewards logically.
reward model preference data preference ranking alignment procedure probabilistic interpretation Bradley-Terry preference models reward transformation poorly-performing outputs reward hacking underfitting aggregated rewards logical conjunction RLHF helpful harmless