title: Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
summary: Merging Large Language Models (LLMs) is a cost-effective technique for
  combining multiple expert LLMs into a single versatile model, retaining the
  expertise of the original ones. However, current approaches often overlook the
  importance of safety alignment during merging, leading to highly misaligned
  models. This work investigates the effects of model merging on alignment. We
  evaluate several popular model merging techniques, demonstrating that existing
  methods do not only transfer domain expertise but also propagate misalignment.
  We propose a simple two-step approach to address this problem: (i) generating
  synthetic safety and domain-specific data, and (ii) incorporating these
  generated data into the optimization process of existing data-aware model
  merging techniques. This allows us to treat alignment as a skill that can be
  maximized in the resulting merged LLM. Our experiments illustrate the
  effectiveness of integrating alignment-related data during merging, resulting
  in models that excel in both domain expertise and alignment.
paper_authors._id: 6675a373a223121b3da870a0
paper_authors.name: Hasan Abed Al Kader Hammoud
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:32:21.606Z
paper_authors._id: 642b51385bf2355d02a23d15
paper_authors.avatarUrl: /avatars/87985347643b2647555f2453fa4d94fb.svg
paper_authors.fullname: Hasan Abed Al Kader Hammoud
paper_authors.type: user
paper_authors.user: hammh0a
paper_authors._id: 6675a373a223121b3da870a1
paper_authors.name: Umberto Michieli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T08:22:59.876Z
paper_authors._id: 646f1879c261dc4133830dd6
paper_authors.avatarUrl: /avatars/f4134a8fb171f93c098a43f1c7a94eb6.svg
paper_authors.fullname: Umberto Michieli
paper_authors.type: user
paper_authors.user: u-michieli
paper_authors._id: 6675a373a223121b3da870a2
paper_authors.name: Fabio Pizzati
paper_authors._id: 6675a373a223121b3da870a3
paper_authors.name: Philip Torr
paper_authors._id: 6675a373a223121b3da870a4
paper_authors.name: Adel Bibi
paper_authors._id: 6675a373a223121b3da870a5
paper_authors.name: Bernard Ghanem
paper_authors._id: 6675a373a223121b3da870a6
paper_authors.name: Mete Ozay
paper_title: Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
paper_summary: Merging Large Language Models (LLMs) is a cost-effective technique for
  combining multiple expert LLMs into a single versatile model, retaining the
  expertise of the original ones. However, current approaches often overlook the
  importance of safety alignment during merging, leading to highly misaligned
  models. This work investigates the effects of model merging on alignment. We
  evaluate several popular model merging techniques, demonstrating that existing
  methods do not only transfer domain expertise but also propagate misalignment.
  We propose a simple two-step approach to address this problem: (i) generating
  synthetic safety and domain-specific data, and (ii) incorporating these
  generated data into the optimization process of existing data-aware model
  merging techniques. This allows us to treat alignment as a skill that can be
  maximized in the resulting merged LLM. Our experiments illustrate the
  effectiveness of integrating alignment-related data during merging, resulting
  in models that excel in both domain expertise and alignment.
paper_ai_summary: Proposed method improves model alignment during the merging of large language models by incorporating synthetic safety and domain-specific data.
paper_ai_keywords: Large Language Models (LLMs) model merging safety alignment domain expertise misalignment synthetic safety data domain-specific data data-aware model merging techniques