title: Activation-Informed Merging of Large Language Models
summary: Model merging, a method that combines the parameters and embeddings of
  multiple fine-tuned large language models (LLMs), offers a promising approach
  to enhance model performance across various tasks while maintaining
  computational efficiency. This paper introduces Activation-Informed Merging
  (AIM), a technique that integrates the information from the activation space of
  LLMs into the merging process to improve performance and robustness. AIM is
  designed as a flexible, complementary solution that is applicable to any
  existing merging method. It aims to preserve critical weights from the base
  model, drawing on principles from continual learning~(CL) and model
  compression. Utilizing a task-agnostic calibration set, AIM selectively
  prioritizes essential weights during merging. We empirically demonstrate that
  AIM significantly enhances the performance of merged models across multiple
  benchmarks. Our findings suggest that considering the activation-space
  information can provide substantial advancements in the model merging
  strategies for LLMs with up to 40\% increase in benchmark performance.
paper_authors._id: 67a4fc2450641c7d60cead58
paper_authors.name: Amin Heyrani Nobari
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-06T18:15:01.447Z
paper_authors._id: 64d516ba80d47a6b76fc1015
paper_authors.avatarUrl: /avatars/e520825f2ac9ff047844496ae2dad7d6.svg
paper_authors.fullname: Amin Heyrani Nobari
paper_authors.type: user
paper_authors.user: ahn1376
paper_authors._id: 67a4fc2450641c7d60cead59
paper_authors.name: Kaveh Alimohammadi
paper_authors._id: 67a4fc2450641c7d60cead5a
paper_authors.name: Ali ArjomandBigdeli
paper_authors._id: 67a4fc2450641c7d60cead5b
paper_authors.name: Akash Srivastava
paper_authors._id: 67a4fc2450641c7d60cead5c
paper_authors.name: Faez Ahmed
paper_authors._id: 67a4fc2450641c7d60cead5d
paper_authors.name: Navid Azizan
paper_title: Activation-Informed Merging of Large Language Models
paper_summary: Model merging, a method that combines the parameters and embeddings of
  multiple fine-tuned large language models (LLMs), offers a promising approach
  to enhance model performance across various tasks while maintaining
  computational efficiency. This paper introduces Activation-Informed Merging
  (AIM), a technique that integrates the information from the activation space of
  LLMs into the merging process to improve performance and robustness. AIM is
  designed as a flexible, complementary solution that is applicable to any
  existing merging method. It aims to preserve critical weights from the base
  model, drawing on principles from continual learning~(CL) and model
  compression. Utilizing a task-agnostic calibration set, AIM selectively
  prioritizes essential weights during merging. We empirically demonstrate that
  AIM significantly enhances the performance of merged models across multiple
  benchmarks. Our findings suggest that considering the activation-space
  information can provide substantial advancements in the model merging
  strategies for LLMs with up to 40\% increase in benchmark performance.
paper_ai_keywords: Contrast embeddings fine-tuned large language models (LLMs) Activation-Informed Merging (AIM) particular space computational efficiency flexible complementary continual learning (CL) model compression task-agnostic calibration set activation-space