title: DELLA-Merging: Reducing Interference in Model Merging through
    Magnitude-Based Sampling
summary: With the proliferation of domain-specific models, model merging has emerged
  as a set of techniques that combine the capabilities of multiple models into
  one that can multitask without the cost of additional training. In this paper,
  we propose a new model merging technique, Drop and rEscaLe via sampLing with
  mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE,
  which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the
  parameters in order of their magnitude and assigns higher dropout probabilities
  (p) to parameters with lower ranks corresponding to lower magnitudes. To
  approximate the original embeddings, MAGPRUNE employs a rescaling operation on
  the parameters that survive the random dropping by 1/(1 - p). On three
  different expert models considered for merging (LM, Math, Code) and
  corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an
  average improvement of 2.4 points over baseline methods employing delta
  parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over
  DARE), and 11.1 points over the no-pruning baseline (TA). We release the source
  code at: https://github.com/declare-lab/della.
paper_authors._id: 66798918e93f2212af0ddb59
paper_authors.name: Pala Tej Deep
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T20:18:15.135Z
paper_authors._id: 633f875450d83f5065d459d6
paper_authors.avatarUrl: /avatars/48312867ac2c05d589869972f90881b9.svg
paper_authors.fullname: Pala Tej Deep
paper_authors.type: user
paper_authors.user: Tej3
paper_authors._id: 66798918e93f2212af0ddb5a
paper_authors.name: Rishabh Bhardwaj
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:08:25.477Z
paper_authors._id: 5f278507e923d665e616271b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
paper_authors.fullname: Rishabh Bhardwaj
paper_authors.type: user
paper_authors.user: RishabhBhardwaj
paper_authors._id: 66798918e93f2212af0ddb5b
paper_authors.name: Soujanya Poria
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:52:37.753Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: DELLA-Merging: Reducing Interference in Model Merging through
    Magnitude-Based Sampling
paper_summary: With the proliferation of domain-specific models, model merging has emerged
  as a set of techniques that combine the capabilities of multiple models into
  one that can multitask without the cost of additional training. In this paper,
  we propose a new model merging technique, Drop and rEscaLe via sampLing with
  mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE,
  which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the
  parameters in order of their magnitude and assigns higher dropout probabilities
  (p) to parameters with lower ranks corresponding to lower magnitudes. To
  approximate the original embeddings, MAGPRUNE employs a rescaling operation on
  the parameters that survive the random dropping by 1/(1 - p). On three
  different expert models considered for merging (LM, Math, Code) and
  corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an
  average improvement of 2.4 points over baseline methods employing delta
  parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over
  DARE), and 11.1 points over the no-pruning baseline (TA). We release the source
  code at: https://github.com/declare-lab/della.
paper_ai_summary: A novel model merging technique, DELLA-Merging, using MAGPRUNE pruning significantly enhances multitasking capabilities of domain-specific models across various benchmarks.
paper_ai_keywords: model merging DELLA-Merging MAGPRUNE pruning parameter magnitude dropout probabilities rescaling expertise models AlpacaEval GSM8K MBPP delta parameter pruning TIES DARE no-pruning baseline