Exploring Model Kinship for Merging Large Language Models
Model merging has become one of the key technologies for enhancing the
capabilities and efficiency of Large Language Models (LLMs). However, our
understanding of the expected performance gains and principles when merging any
two models remains limited. In this work, we introduce model kinship, the
degree of similarity or relatedness between LLMs, analogous to biological
evolution. With comprehensive empirical analysis, we find that there is a
certain relationship between model kinship and the performance gains after
model merging, which can help guide our selection of candidate models. Inspired
by this, we propose a new model merging strategy: Top-k Greedy Merging with
Model Kinship, which can yield better performance on benchmark datasets.
Specifically, we discover that using model kinship as a criterion can assist us
in continuously performing model merging, alleviating the degradation (local
optima) in model evolution, whereas model kinship can serve as a guide to
escape these traps. Code is available at
https://github.com/zjunlp/ModelKinship.
6710757f66dbb164b08db625
Yedi Hu
claimed_verified
2024-10-22T09:33:01.417Z
660a25825a3d57c6c5eb57d7
https://cdn-avatars.huggingface.co/v1/production/uploads/660a25825a3d57c6c5eb57d7/LUC2-egzwaNuGuW-do33N.png
Yedi Hu
user
yedi-hu
6710757f66dbb164b08db626
Yunzhi Yao
6710757f66dbb164b08db627
Ningyu Zhang
claimed_verified
2024-10-18T08:25:06.237Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
6710757f66dbb164b08db628
Shumin Deng
6710757f66dbb164b08db629
Huajun Chen
Exploring Model Kinship for Merging Large Language Models
Model merging has become one of the key technologies for enhancing the
capabilities and efficiency of Large Language Models (LLMs). However, our
understanding of the expected performance gains and principles when merging any
two models remains limited. In this work, we introduce model kinship, the
degree of similarity or relatedness between LLMs, analogous to biological
evolution. With comprehensive empirical analysis, we find that there is a
certain relationship between model kinship and the performance gains after
model merging, which can help guide our selection of candidate models. Inspired
by this, we propose a new model merging strategy: Top-k Greedy Merging with
Model Kinship, which can yield better performance on benchmark datasets.
Specifically, we discover that using model kinship as a criterion can assist us
in continuously performing model merging, alleviating the degradation (local
optima) in model evolution, whereas model kinship can serve as a guide to
escape these traps. Code is available at
https://github.com/zjunlp/ModelKinship.
Model kinship guides model merging to improve performance on benchmarks and helps escape local optima in model evolution.
Large Language Models (LLMs) model merging model kinship biological evolution top-k Greedy Merging with Model Kinship