title: Evolutionary Optimization of Model Merging Recipes
summary: We present a novel application of evolutionary algorithms to automate the
  creation of powerful foundation models. While model merging has emerged as a
  promising approach for LLM development due to its cost-effectiveness, it
  currently relies on human intuition and domain knowledge, limiting its
  potential. Here, we propose an evolutionary approach that overcomes this
  limitation by automatically discovering effective combinations of diverse
  open-source models, harnessing their collective intelligence without requiring
  extensive additional training data or compute. Our approach operates in both
  parameter space and data flow space, allowing for optimization beyond just the
  weights of the individual models. This approach even facilitates cross-domain
  merging, generating models like a Japanese LLM with Math reasoning
  capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art
  performance on a variety of established Japanese LLM benchmarks, even
  surpassing models with significantly more parameters, despite not being
  explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM
  generated through our approach demonstrates its effectiveness in describing
  Japanese culture-specific content, outperforming previous Japanese VLMs. This
  work not only contributes new state-of-the-art models back to the open-source
  community, but also introduces a new paradigm for automated model composition,
  paving the way for exploring alternative, efficient approaches to foundation
  model development.
paper_authors._id: 65fb7f71ec1f99ad1f01f4ea
paper_authors.name: Takuya Akiba
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:07:28.747Z
paper_authors._id: 6482810dba6c556892f6f257
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
paper_authors.fullname: Takuya Akiba
paper_authors.type: user
paper_authors.user: iwiwi
paper_authors._id: 65fb7f71ec1f99ad1f01f4eb
paper_authors.name: Makoto Shing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:26:55.128Z
paper_authors._id: 60c2e7747a42b2edc5d2ccf7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60c2e7747a42b2edc5d2ccf7/5GdUBW1HYEy17orItOqfV.png
paper_authors.fullname: Makoto Shing
paper_authors.type: user
paper_authors.user: mkshing
paper_authors._id: 65fb7f71ec1f99ad1f01f4ec
paper_authors.name: Yujin Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:07:55.475Z
paper_authors._id: 63285beda4102056bc198395
paper_authors.avatarUrl: /avatars/6c936818a3814c1c32a8eaae8b8cb6d6.svg
paper_authors.fullname: yujin tang
paper_authors.type: user
paper_authors.user: tyj2022
paper_authors._id: 65fb7f71ec1f99ad1f01f4ed
paper_authors.name: Qi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T10:02:42.615Z
paper_authors._id: 644b983f0fbe4830f192c4f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644b983f0fbe4830f192c4f5/4eX4OS0gnXGqLu3eaMml_.png
paper_authors.fullname: Qi Sun
paper_authors.type: user
paper_authors.user: lfsm
paper_authors._id: 65fb7f71ec1f99ad1f01f4ee
paper_authors.name: David Ha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:08:27.543Z
paper_authors._id: 6335ea47b5a0e02f80538bf3
paper_authors.avatarUrl: /avatars/fce4e08ae86339952e35ff73680da7e9.svg
paper_authors.fullname: David
paper_authors.type: user
paper_authors.user: davidha
paper_title: Evolutionary Optimization of Model Merging Recipes
paper_summary: We present a novel application of evolutionary algorithms to automate the
  creation of powerful foundation models. While model merging has emerged as a
  promising approach for LLM development due to its cost-effectiveness, it
  currently relies on human intuition and domain knowledge, limiting its
  potential. Here, we propose an evolutionary approach that overcomes this
  limitation by automatically discovering effective combinations of diverse
  open-source models, harnessing their collective intelligence without requiring
  extensive additional training data or compute. Our approach operates in both
  parameter space and data flow space, allowing for optimization beyond just the
  weights of the individual models. This approach even facilitates cross-domain
  merging, generating models like a Japanese LLM with Math reasoning
  capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art
  performance on a variety of established Japanese LLM benchmarks, even
  surpassing models with significantly more parameters, despite not being
  explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM
  generated through our approach demonstrates its effectiveness in describing
  Japanese culture-specific content, outperforming previous Japanese VLMs. This
  work not only contributes new state-of-the-art models back to the open-source
  community, but also introduces a new paradigm for automated model composition,
  paving the way for exploring alternative, efficient approaches to foundation
  model development.
paper_ai_keywords: evolutionary algorithms foundation models model merging LLM development cost-effectiveness cross-attention layers parameter space data flow space cross-domain merging Japanese LLM Math reasoning capabilities Japanese LLM benchmarks culturally-aware Japanese VLM automated model composition