title: Exploring Model Kinship for Merging Large Language Models
summary: Model merging has become one of the key technologies for enhancing the
  capabilities and efficiency of Large Language Models (LLMs). However, our
  understanding of the expected performance gains and principles when merging any
  two models remains limited. In this work, we introduce model kinship, the
  degree of similarity or relatedness between LLMs, analogous to biological
  evolution. With comprehensive empirical analysis, we find that there is a
  certain relationship between model kinship and the performance gains after
  model merging, which can help guide our selection of candidate models. Inspired
  by this, we propose a new model merging strategy: Top-k Greedy Merging with
  Model Kinship, which can yield better performance on benchmark datasets.
  Specifically, we discover that using model kinship as a criterion can assist us
  in continuously performing model merging, alleviating the degradation (local
  optima) in model evolution, whereas model kinship can serve as a guide to
  escape these traps. Code is available at
  https://github.com/zjunlp/ModelKinship.
paper_authors._id: 6710757f66dbb164b08db625
paper_authors.name: Yedi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T09:33:01.417Z
paper_authors._id: 660a25825a3d57c6c5eb57d7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/660a25825a3d57c6c5eb57d7/LUC2-egzwaNuGuW-do33N.png
paper_authors.fullname: Yedi Hu
paper_authors.type: user
paper_authors.user: yedi-hu
paper_authors._id: 6710757f66dbb164b08db626
paper_authors.name: Yunzhi Yao
paper_authors._id: 6710757f66dbb164b08db627
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:25:06.237Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_authors._id: 6710757f66dbb164b08db628
paper_authors.name: Shumin Deng
paper_authors._id: 6710757f66dbb164b08db629
paper_authors.name: Huajun Chen
paper_title: Exploring Model Kinship for Merging Large Language Models
paper_summary: Model merging has become one of the key technologies for enhancing the
  capabilities and efficiency of Large Language Models (LLMs). However, our
  understanding of the expected performance gains and principles when merging any
  two models remains limited. In this work, we introduce model kinship, the
  degree of similarity or relatedness between LLMs, analogous to biological
  evolution. With comprehensive empirical analysis, we find that there is a
  certain relationship between model kinship and the performance gains after
  model merging, which can help guide our selection of candidate models. Inspired
  by this, we propose a new model merging strategy: Top-k Greedy Merging with
  Model Kinship, which can yield better performance on benchmark datasets.
  Specifically, we discover that using model kinship as a criterion can assist us
  in continuously performing model merging, alleviating the degradation (local
  optima) in model evolution, whereas model kinship can serve as a guide to
  escape these traps. Code is available at
  https://github.com/zjunlp/ModelKinship.
paper_ai_summary: Model kinship guides model merging to improve performance on benchmarks and helps escape local optima in model evolution.
paper_ai_keywords: Large Language Models (LLMs) model merging model kinship biological evolution top-k Greedy Merging with Model Kinship