WARM: On the Benefits of Weight Averaged Reward Models
Aligning large language models (LLMs) with human preferences through
reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit
failures in the reward model (RM) to achieve seemingly high rewards without
meeting the underlying objectives. We identify two primary challenges when
designing RMs to mitigate reward hacking: distribution shifts during the RL
process and inconsistencies in human preferences. As a solution, we propose
Weight Averaged Reward Models (WARM), first fine-tuning multiple RMs, then
averaging them in the weight space. This strategy follows the observation that
fine-tuned weights remain linearly mode connected when sharing the same
pre-training. By averaging weights, WARM improves efficiency compared to the
traditional ensembling of predictions, while improving reliability under
distribution shifts and robustness to preference inconsistencies. Our
experiments on summarization tasks, using best-of-N and RL methods, shows that
WARM improves the overall quality and alignment of LLM predictions; for
example, a policy RL fine-tuned with WARM has a 79.4% win rate against a policy
RL fine-tuned with a single RM.
65af3c851f418c7449389cc1
Alexandre Ramé
admin_assigned
2024-01-23T10:48:42.494Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
65af3c851f418c7449389cc2
Nino Vieillard
65af3c851f418c7449389cc3
Léonard Hussenot
claimed_verified
2025-03-28T14:22:47.080Z
667a7b2e1764931987803b46
/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
Léonard Hussenot
user
leonardhussenot
65af3c851f418c7449389cc4
Robert Dadashi
65af3c851f418c7449389cc5
Geoffrey Cideron
65af3c851f418c7449389cc6
Olivier Bachem
65af3c851f418c7449389cc7
Johan Ferret
claimed_verified
2024-01-23T13:48:51.074Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
WARM: On the Benefits of Weight Averaged Reward Models
Aligning large language models (LLMs) with human preferences through
reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit
failures in the reward model (RM) to achieve seemingly high rewards without
meeting the underlying objectives. We identify two primary challenges when
designing RMs to mitigate reward hacking: distribution shifts during the RL
process and inconsistencies in human preferences. As a solution, we propose
Weight Averaged Reward Models (WARM), first fine-tuning multiple RMs, then
averaging them in the weight space. This strategy follows the observation that
fine-tuned weights remain linearly mode connected when sharing the same
pre-training. By averaging weights, WARM improves efficiency compared to the
traditional ensembling of predictions, while improving reliability under
distribution shifts and robustness to preference inconsistencies. Our
experiments on summarization tasks, using best-of-N and RL methods, shows that
WARM improves the overall quality and alignment of LLM predictions; for
example, a policy RL fine-tuned with WARM has a 79.4% win rate against a policy
RL fine-tuned with a single RM.
Weight Averaged Reward Models (WARM) improve the alignment and quality of large language models by averaging fine-tuned reward models, addressing distribution shifts and preference inconsistencies.
large language models reinforcement learning RLHF reward hacking reward model distribution shifts human preferences Weight Averaged Reward Models WARM fine-tuning pre-training linearly mode connected ensembling summarization tasks policy RL