title: No Task Left Behind: Isotropic Model Merging with Common and
    Task-Specific Subspaces
summary: Model merging integrates the weights of multiple task-specific models into a
  single multi-task model. Despite recent interest in the problem, a significant
  performance gap between the combined and single-task models remains. In this
  paper, we investigate the key characteristics of task matrices -- weight update
  matrices applied to a pre-trained model -- that enable effective merging. We
  show that alignment between singular components of task-specific and merged
  matrices strongly correlates with performance improvement over the pre-trained
  model. Based on this, we propose an isotropic merging framework that flattens
  the singular value spectrum of task matrices, enhances alignment, and reduces
  the performance gap. Additionally, we incorporate both common and task-specific
  subspaces to further improve alignment and performance. Our proposed approach
  achieves state-of-the-art performance across multiple scenarios, including
  various sets of tasks and model scales. This work advances the understanding of
  model merging dynamics, offering an effective methodology to merge models
  without requiring additional training. Code is available at
  https://github.com/danielm1405/iso-merging .
paper_authors._id: 67a9f4900b97667e0a82ad3d
paper_authors.name: Daniel Marczak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T13:11:49.393Z
paper_authors._id: 65a5358ddb5c00652ef24c8d
paper_authors.avatarUrl: /avatars/d50b6297584c9b4c2ccd93e64477b940.svg
paper_authors.fullname: Daniel Marczak
paper_authors.type: user
paper_authors.user: danielm1405
paper_authors._id: 67a9f4900b97667e0a82ad3e
paper_authors.name: Simone Magistri
paper_authors._id: 67a9f4900b97667e0a82ad3f
paper_authors.name: Sebastian Cygert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:17:02.946Z
paper_authors._id: 6763f031c0a39e58c57ed9f9
paper_authors.avatarUrl: /avatars/2e864838a571f52b5316f90d60b763f1.svg
paper_authors.fullname: Sebastian Cygert
paper_authors.type: user
paper_authors.user: cygerts
paper_authors._id: 67a9f4900b97667e0a82ad40
paper_authors.name: Bartłomiej Twardowski
paper_authors._id: 67a9f4900b97667e0a82ad41
paper_authors.name: Andrew D. Bagdanov
paper_authors._id: 67a9f4900b97667e0a82ad42
paper_authors.name: Joost van de Weijer
paper_title: No Task Left Behind: Isotropic Model Merging with Common and
    Task-Specific Subspaces
paper_summary: Model merging integrates the weights of multiple task-specific models into a
  single multi-task model. Despite recent interest in the problem, a significant
  performance gap between the combined and single-task models remains. In this
  paper, we investigate the key characteristics of task matrices -- weight update
  matrices applied to a pre-trained model -- that enable effective merging. We
  show that alignment between singular components of task-specific and merged
  matrices strongly correlates with performance improvement over the pre-trained
  model. Based on this, we propose an isotropic merging framework that flattens
  the singular value spectrum of task matrices, enhances alignment, and reduces
  the performance gap. Additionally, we incorporate both common and task-specific
  subspaces to further improve alignment and performance. Our proposed approach
  achieves state-of-the-art performance across multiple scenarios, including
  various sets of tasks and model scales. This work advances the understanding of
  model merging dynamics, offering an effective methodology to merge models
  without requiring additional training. Code is available at
  https://github.com/danielm1405/iso-merging .
paper_ai_summary: The paper proposes isotropic merging to improve performance during model merging by aligning singular components and incorporating common and task-specific subspaces.
paper_ai_keywords: task matrices weight update matrices singular components isotropic merging singular value spectrum common subspaces task-specific subspaces