title: WARP: On the Benefits of Weight Averaged Rewarded Policies
summary: Reinforcement learning from human feedback (RLHF) aligns large language
  models (LLMs) by encouraging their generations to have high rewards, using a
  reward model trained on human preferences. To prevent the forgetting of
  pre-trained knowledge, RLHF usually incorporates a KL regularization; this
  forces the policy to remain close to its supervised fine-tuned initialization,
  though it hinders the reward optimization. To tackle the trade-off between KL
  and reward, in this paper we introduce a novel alignment strategy named Weight
  Averaged Rewarded Policies (WARP). WARP merges policies in the weight space at
  three distinct stages. First, it uses the exponential moving average of the
  policy as a dynamic anchor in the KL regularization. Second, it applies
  spherical interpolation to merge independently fine-tuned policies into a new
  enhanced one. Third, it linearly interpolates between this merged model and the
  initialization, to recover features from pre-training. This procedure is then
  applied iteratively, with each iteration's final model used as an advanced
  initialization for the next, progressively refining the KL-reward Pareto front,
  achieving superior rewards at fixed KL. Experiments with GEMMA policies
  validate that WARP improves their quality and alignment, outperforming other
  open-source LLMs.
paper_authors._id: 667a598613c37a0fe4a8b06e
paper_authors.name: Alexandre Ramé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:17:55.519Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 667a598613c37a0fe4a8b06f
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:18:03.169Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 667a598613c37a0fe4a8b070
paper_authors.name: Nino Vieillard
paper_authors._id: 667a598613c37a0fe4a8b071
paper_authors.name: Robert Dadashi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:18:17.076Z
paper_authors._id: 65fc7c0da57b88a765aea493
paper_authors.avatarUrl: /avatars/440f88769da1113d6158fe7e0514ead3.svg
paper_authors.fullname: Robert Dadashi
paper_authors.type: user
paper_authors.user: ddsh
paper_authors._id: 667a598613c37a0fe4a8b072
paper_authors.name: Léonard Hussenot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:18:24.900Z
paper_authors._id: 667a7b2e1764931987803b46
paper_authors.avatarUrl: /avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
paper_authors.fullname: Léonard Hussenot
paper_authors.type: user
paper_authors.user: leonardhussenot
paper_authors._id: 667a598613c37a0fe4a8b073
paper_authors.name: Pierre-Louis Cedoz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:18:30.765Z
paper_authors._id: 64dbdd39e7bc8544f9ac4c1a
paper_authors.avatarUrl: /avatars/7aface0db75500a2b0ede75179be35ce.svg
paper_authors.fullname: Pierre-Louis Cedoz
paper_authors.type: user
paper_authors.user: plcedoz
paper_authors._id: 667a598613c37a0fe4a8b074
paper_authors.name: Pier Giuseppe Sessa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:18:37.543Z
paper_authors._id: 667a7876becec8fc5112cf1f
paper_authors.avatarUrl: /avatars/6874f0e5a7edca4145c57d7908f89840.svg
paper_authors.fullname: Pier Giuseppe Sessa
paper_authors.type: user
paper_authors.user: piergs
paper_authors._id: 667a598613c37a0fe4a8b075
paper_authors.name: Sertan Girgin
paper_authors._id: 667a598613c37a0fe4a8b076
paper_authors.name: Arthur Douillard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:32:35.814Z
paper_authors._id: 622792366303bf1dc304f49f
paper_authors.avatarUrl: /avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
paper_authors.fullname: Arthur Douillard
paper_authors.type: user
paper_authors.user: ArthurDouillard
paper_authors._id: 667a598613c37a0fe4a8b077
paper_authors.name: Olivier Bachem
paper_title: WARP: On the Benefits of Weight Averaged Rewarded Policies
paper_summary: Reinforcement learning from human feedback (RLHF) aligns large language
  models (LLMs) by encouraging their generations to have high rewards, using a
  reward model trained on human preferences. To prevent the forgetting of
  pre-trained knowledge, RLHF usually incorporates a KL regularization; this
  forces the policy to remain close to its supervised fine-tuned initialization,
  though it hinders the reward optimization. To tackle the trade-off between KL
  and reward, in this paper we introduce a novel alignment strategy named Weight
  Averaged Rewarded Policies (WARP). WARP merges policies in the weight space at
  three distinct stages. First, it uses the exponential moving average of the
  policy as a dynamic anchor in the KL regularization. Second, it applies
  spherical interpolation to merge independently fine-tuned policies into a new
  enhanced one. Third, it linearly interpolates between this merged model and the
  initialization, to recover features from pre-training. This procedure is then
  applied iteratively, with each iteration's final model used as an advanced
  initialization for the next, progressively refining the KL-reward Pareto front,
  achieving superior rewards at fixed KL. Experiments with GEMMA policies
  validate that WARP improves their quality and alignment, outperforming other
  open-source LLMs.
paper_ai_keywords: Reinforcement learning from human feedback (RLHF) large language models (LLMs) reward optimization KL regularization Weight Averaged Rewarded Policies (WARP) exponential moving average spherical interpolation linear interpolation KL-reward Pareto front GEMMA policies