Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
Merging Large Language Models (LLMs) is a cost-effective technique for
combining multiple expert LLMs into a single versatile model, retaining the
expertise of the original ones. However, current approaches often overlook the
importance of safety alignment during merging, leading to highly misaligned
models. This work investigates the effects of model merging on alignment. We
evaluate several popular model merging techniques, demonstrating that existing
methods do not only transfer domain expertise but also propagate misalignment.
We propose a simple two-step approach to address this problem: (i) generating
synthetic safety and domain-specific data, and (ii) incorporating these
generated data into the optimization process of existing data-aware model
merging techniques. This allows us to treat alignment as a skill that can be
maximized in the resulting merged LLM. Our experiments illustrate the
effectiveness of integrating alignment-related data during merging, resulting
in models that excel in both domain expertise and alignment.
6675a373a223121b3da870a0
Hasan Abed Al Kader Hammoud
claimed_verified
2024-06-24T07:32:21.606Z
642b51385bf2355d02a23d15
/avatars/87985347643b2647555f2453fa4d94fb.svg
Hasan Abed Al Kader Hammoud
user
hammh0a
6675a373a223121b3da870a1
Umberto Michieli
claimed_verified
2024-12-12T08:22:59.876Z
646f1879c261dc4133830dd6
/avatars/f4134a8fb171f93c098a43f1c7a94eb6.svg
Umberto Michieli
user
u-michieli
6675a373a223121b3da870a2
Fabio Pizzati
6675a373a223121b3da870a3
Philip Torr
6675a373a223121b3da870a4
Adel Bibi
6675a373a223121b3da870a5
Bernard Ghanem
6675a373a223121b3da870a6
Mete Ozay
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
Merging Large Language Models (LLMs) is a cost-effective technique for
combining multiple expert LLMs into a single versatile model, retaining the
expertise of the original ones. However, current approaches often overlook the
importance of safety alignment during merging, leading to highly misaligned
models. This work investigates the effects of model merging on alignment. We
evaluate several popular model merging techniques, demonstrating that existing
methods do not only transfer domain expertise but also propagate misalignment.
We propose a simple two-step approach to address this problem: (i) generating
synthetic safety and domain-specific data, and (ii) incorporating these
generated data into the optimization process of existing data-aware model
merging techniques. This allows us to treat alignment as a skill that can be
maximized in the resulting merged LLM. Our experiments illustrate the
effectiveness of integrating alignment-related data during merging, resulting
in models that excel in both domain expertise and alignment.
Proposed method improves model alignment during the merging of large language models by incorporating synthetic safety and domain-specific data.
Large Language Models (LLMs) model merging safety alignment domain expertise misalignment synthetic safety data domain-specific data data-aware model merging techniques