title: SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
summary: In Large Language Model (LLM) development, Reinforcement Learning from Human
  Feedback (RLHF) is crucial for aligning models with human values and
  preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence
  between the current policy and a frozen initial policy as a reference, which is
  added as a penalty in policy optimization algorithms like Proximal Policy
  Optimization (PPO). While this constraint prevents models from deviating too
  far from the initial checkpoint, it limits exploration of the reward landscape,
  reducing the model's ability to discover higher-quality solutions. As a result,
  policy optimization is often trapped in a narrow region of the parameter space,
  leading to suboptimal alignment and performance. This paper presents SALSA
  (Soup-based Alignment Learning for Stronger Adaptation), a novel approach
  designed to overcome these limitations by creating a more flexible and better
  located reference model through weight-space averaging of two independent
  supervised fine-tuned (SFT) models. This model soup allows for larger deviation
  in KL divergence and exploring a promising region of the solution space without
  sacrificing stability. By leveraging this more robust reference model, SALSA
  fosters better exploration, achieving higher rewards and improving model
  robustness, out-of-distribution generalization, and performance. We validate
  the effectiveness of SALSA through extensive experiments on popular open models
  (Llama2-7B, Mistral-7B, and Gemma-2B) across various benchmarks (MT-Bench,
  Arena-Hard, UltraFeedback), where it consistently surpasses PPO by fostering
  deeper exploration and achieving superior alignment in LLMs.
paper_authors._id: 672a8daf673a4e2d70053a50
paper_authors.name: Atoosa Chegini
paper_authors._id: 672a8daf673a4e2d70053a51
paper_authors.name: Hamid Kazemi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:37:35.271Z
paper_authors._id: 64f1f9612820a6f1b9e19edd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f1f9612820a6f1b9e19edd/hSmhLsczKR1t5jxHwEaE_.jpeg
paper_authors.fullname: Hamid Kazemi
paper_authors.type: user
paper_authors.user: seyedhamidreza
paper_authors._id: 672a8daf673a4e2d70053a52
paper_authors.name: Iman Mirzadeh
paper_authors._id: 672a8daf673a4e2d70053a53
paper_authors.name: Dong Yin
paper_authors._id: 672a8daf673a4e2d70053a54
paper_authors.name: Maxwell Horton
paper_authors._id: 672a8daf673a4e2d70053a55
paper_authors.name: Moin Nabi
paper_authors._id: 672a8daf673a4e2d70053a56
paper_authors.name: Mehrdad Farajtabar
paper_authors._id: 672a8daf673a4e2d70053a57
paper_authors.name: Keivan Alizadeh
paper_title: SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
paper_summary: In Large Language Model (LLM) development, Reinforcement Learning from Human
  Feedback (RLHF) is crucial for aligning models with human values and
  preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence
  between the current policy and a frozen initial policy as a reference, which is
  added as a penalty in policy optimization algorithms like Proximal Policy
  Optimization (PPO). While this constraint prevents models from deviating too
  far from the initial checkpoint, it limits exploration of the reward landscape,
  reducing the model's ability to discover higher-quality solutions. As a result,
  policy optimization is often trapped in a narrow region of the parameter space,
  leading to suboptimal alignment and performance. This paper presents SALSA
  (Soup-based Alignment Learning for Stronger Adaptation), a novel approach
  designed to overcome these limitations by creating a more flexible and better
  located reference model through weight-space averaging of two independent
  supervised fine-tuned (SFT) models. This model soup allows for larger deviation
  in KL divergence and exploring a promising region of the solution space without
  sacrificing stability. By leveraging this more robust reference model, SALSA
  fosters better exploration, achieving higher rewards and improving model
  robustness, out-of-distribution generalization, and performance. We validate
  the effectiveness of SALSA through extensive experiments on popular open models
  (Llama2-7B, Mistral-7B, and Gemma-2B) across various benchmarks (MT-Bench,
  Arena-Hard, UltraFeedback), where it consistently surpasses PPO by fostering
  deeper exploration and achieving superior alignment in LLMs.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Kullback-Leibler (KL) divergence Proximal Policy Optimization (PPO) policy optimization algorithms parameter space Visual Question Answering weight-space averaging supervised fine-tuned (SFT) model soup out-of-distribution generalization MT-Bench Arena-Hard UltraFeedback vela Llama2-7B Mistral-7B Gemma-2B vela vela SALSA (Soup-based Alignment Learning for Stronger Adaptation)