title: What Matters for Model Merging at Scale?
summary: Model merging aims to combine multiple expert models into a more capable
  single model, offering benefits such as reduced storage and serving costs,
  improved generalization, and support for decentralized model development.
  Despite its promise, previous studies have primarily focused on merging a few
  small models. This leaves many unanswered questions about the effect of scaling
  model size and how it interplays with other key factors -- like the base model
  quality and number of expert models -- , to affect the merged model's
  performance. This work systematically evaluates the utility of model merging at
  scale, examining the impact of these different factors. We experiment with
  merging fully fine-tuned models using 4 popular merging methods -- Averaging,
  Task~Arithmetic, Dare, and TIES -- across model sizes ranging from 1B-64B
  parameters and merging up to 8 different expert models. We evaluate the merged
  models on both held-in tasks, i.e., the expert's training tasks, and zero-shot
  generalization to unseen held-out tasks. Our experiments provide several new
  insights about model merging at scale and the interplay between different
  factors. First, we find that merging is more effective when experts are created
  from strong base models, i.e., models with good zero-shot performance. Second,
  larger models facilitate easier merging. Third merging consistently improves
  generalization capabilities. Notably, when merging 8 large expert models, the
  merged models often generalize better compared to the multitask trained models.
  Fourth, we can better merge more expert models when working with larger models.
  Fifth, different merging methods behave very similarly at larger scales.
  Overall, our findings shed light on some interesting properties of model
  merging while also highlighting some limitations. We hope that this study will
  serve as a reference point on large-scale merging for upcoming research.
paper_authors._id: 67053aeed2d7aab4e5ad2d51
paper_authors.name: Prateek Yadav
paper_authors._id: 67053aeed2d7aab4e5ad2d52
paper_authors.name: Tu Vu
paper_authors._id: 67053aeed2d7aab4e5ad2d53
paper_authors.name: Jonathan Lai
paper_authors._id: 67053aeed2d7aab4e5ad2d54
paper_authors.name: Alexandra Chronopoulou
paper_authors._id: 67053aeed2d7aab4e5ad2d55
paper_authors.name: Manaal Faruqui
paper_authors._id: 67053aeed2d7aab4e5ad2d56
paper_authors.name: Mohit Bansal
paper_authors._id: 67053aeed2d7aab4e5ad2d57
paper_authors.name: Tsendsuren Munkhdalai
paper_title: What Matters for Model Merging at Scale?
paper_summary: Model merging aims to combine multiple expert models into a more capable
  single model, offering benefits such as reduced storage and serving costs,
  improved generalization, and support for decentralized model development.
  Despite its promise, previous studies have primarily focused on merging a few
  small models. This leaves many unanswered questions about the effect of scaling
  model size and how it interplays with other key factors -- like the base model
  quality and number of expert models -- , to affect the merged model's
  performance. This work systematically evaluates the utility of model merging at
  scale, examining the impact of these different factors. We experiment with
  merging fully fine-tuned models using 4 popular merging methods -- Averaging,
  Task~Arithmetic, Dare, and TIES -- across model sizes ranging from 1B-64B
  parameters and merging up to 8 different expert models. We evaluate the merged
  models on both held-in tasks, i.e., the expert's training tasks, and zero-shot
  generalization to unseen held-out tasks. Our experiments provide several new
  insights about model merging at scale and the interplay between different
  factors. First, we find that merging is more effective when experts are created
  from strong base models, i.e., models with good zero-shot performance. Second,
  larger models facilitate easier merging. Third merging consistently improves
  generalization capabilities. Notably, when merging 8 large expert models, the
  merged models often generalize better compared to the multitask trained models.
  Fourth, we can better merge more expert models when working with larger models.
  Fifth, different merging methods behave very similarly at larger scales.
  Overall, our findings shed light on some interesting properties of model
  merging while also highlighting some limitations. We hope that this study will
  serve as a reference point on large-scale merging for upcoming research.
paper_ai_summary: The study evaluates various model merging techniques at scale, demonstrating that merging high-quality, large models improves generalization across different tasks.
paper_ai_keywords: model merging fully fine-tuned models Averaging Task Arithmetic Dare TIES parameters zero-shot generalization base model quality generalization capabilities multitask trained models