title: WARM: On the Benefits of Weight Averaged Reward Models
summary: Aligning large language models (LLMs) with human preferences through
  reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit
  failures in the reward model (RM) to achieve seemingly high rewards without
  meeting the underlying objectives. We identify two primary challenges when
  designing RMs to mitigate reward hacking: distribution shifts during the RL
  process and inconsistencies in human preferences. As a solution, we propose
  Weight Averaged Reward Models (WARM), first fine-tuning multiple RMs, then
  averaging them in the weight space. This strategy follows the observation that
  fine-tuned weights remain linearly mode connected when sharing the same
  pre-training. By averaging weights, WARM improves efficiency compared to the
  traditional ensembling of predictions, while improving reliability under
  distribution shifts and robustness to preference inconsistencies. Our
  experiments on summarization tasks, using best-of-N and RL methods, shows that
  WARM improves the overall quality and alignment of LLM predictions; for
  example, a policy RL fine-tuned with WARM has a 79.4% win rate against a policy
  RL fine-tuned with a single RM.
paper_authors._id: 65af3c851f418c7449389cc1
paper_authors.name: Alexandre Ramé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T10:48:42.494Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 65af3c851f418c7449389cc2
paper_authors.name: Nino Vieillard
paper_authors._id: 65af3c851f418c7449389cc3
paper_authors.name: Léonard Hussenot
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T14:22:47.080Z
paper_authors._id: 667a7b2e1764931987803b46
paper_authors.avatarUrl: /avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
paper_authors.fullname: Léonard Hussenot
paper_authors.type: user
paper_authors.user: leonardhussenot
paper_authors._id: 65af3c851f418c7449389cc4
paper_authors.name: Robert Dadashi
paper_authors._id: 65af3c851f418c7449389cc5
paper_authors.name: Geoffrey Cideron
paper_authors._id: 65af3c851f418c7449389cc6
paper_authors.name: Olivier Bachem
paper_authors._id: 65af3c851f418c7449389cc7
paper_authors.name: Johan Ferret
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T13:48:51.074Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_title: WARM: On the Benefits of Weight Averaged Reward Models
paper_summary: Aligning large language models (LLMs) with human preferences through
  reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit
  failures in the reward model (RM) to achieve seemingly high rewards without
  meeting the underlying objectives. We identify two primary challenges when
  designing RMs to mitigate reward hacking: distribution shifts during the RL
  process and inconsistencies in human preferences. As a solution, we propose
  Weight Averaged Reward Models (WARM), first fine-tuning multiple RMs, then
  averaging them in the weight space. This strategy follows the observation that
  fine-tuned weights remain linearly mode connected when sharing the same
  pre-training. By averaging weights, WARM improves efficiency compared to the
  traditional ensembling of predictions, while improving reliability under
  distribution shifts and robustness to preference inconsistencies. Our
  experiments on summarization tasks, using best-of-N and RL methods, shows that
  WARM improves the overall quality and alignment of LLM predictions; for
  example, a policy RL fine-tuned with WARM has a 79.4% win rate against a policy
  RL fine-tuned with a single RM.
paper_ai_summary: Weight Averaged Reward Models (WARM) improve the alignment and quality of large language models by averaging fine-tuned reward models, addressing distribution shifts and preference inconsistencies.
paper_ai_keywords: large language models reinforcement learning RLHF reward hacking reward model distribution shifts human preferences Weight Averaged Reward Models WARM fine-tuning pre-training linearly mode connected ensembling summarization tasks policy RL