title: Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and
    Uncertainty Based Routing
summary: Model merging aggregates Large Language Models (LLMs) finetuned on different
  tasks into a stronger one. However, parameter conflicts between models leads to
  performance degradation in averaging. While model routing addresses this issue
  by selecting individual models during inference, it imposes excessive storage
  and compute costs, and fails to leverage the common knowledge from different
  models. In this work, we observe that different layers exhibit varying levels
  of parameter conflicts. Building on this insight, we average layers with
  minimal parameter conflicts and use a novel task-level expert routing for
  layers with significant conflicts. To further reduce storage costs, inspired by
  task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense
  expert and several sparse experts. Considering the out-of-distribution samples,
  we select and merge appropriate experts based on the task uncertainty of the
  input data. We conduct extensive experiments on both LLaMA and Qwen with
  varying parameter scales, and evaluate on real-world reasoning tasks. Results
  demonstrate that our method consistently achieves significant performance
  improvements while requiring less system cost compared to existing methods.
paper_authors._id: 67adad972883187d78409a7a
paper_authors.name: Kunfeng Lai
paper_authors._id: 67adad972883187d78409a7b
paper_authors.name: Zhenheng Tang
paper_authors._id: 67adad972883187d78409a7c
paper_authors.name: Xinglin Pan
paper_authors._id: 67adad972883187d78409a7d
paper_authors.name: Peijie Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:53:30.947Z
paper_authors._id: 6395f845aec00abff778ad31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
paper_authors.fullname: PeijieDong
paper_authors.type: user
paper_authors.user: pprp
paper_authors._id: 67adad972883187d78409a7e
paper_authors.name: Xiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:45:17.030Z
paper_authors._id: 63024676056ec3a2a8714b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
paper_authors.fullname: Xiang Liu
paper_authors.type: user
paper_authors.user: Dominic789654
paper_authors._id: 67adad972883187d78409a7f
paper_authors.name: Haolan Chen
paper_authors._id: 67adad972883187d78409a80
paper_authors.name: Li Shen
paper_authors._id: 67adad972883187d78409a81
paper_authors.name: Bo Li
paper_authors._id: 67adad972883187d78409a82
paper_authors.name: Xiaowen Chu
paper_title: Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and
    Uncertainty Based Routing
paper_summary: Model merging aggregates Large Language Models (LLMs) finetuned on different
  tasks into a stronger one. However, parameter conflicts between models leads to
  performance degradation in averaging. While model routing addresses this issue
  by selecting individual models during inference, it imposes excessive storage
  and compute costs, and fails to leverage the common knowledge from different
  models. In this work, we observe that different layers exhibit varying levels
  of parameter conflicts. Building on this insight, we average layers with
  minimal parameter conflicts and use a novel task-level expert routing for
  layers with significant conflicts. To further reduce storage costs, inspired by
  task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense
  expert and several sparse experts. Considering the out-of-distribution samples,
  we select and merge appropriate experts based on the task uncertainty of the
  input data. We conduct extensive experiments on both LLaMA and Qwen with
  varying parameter scales, and evaluate on real-world reasoning tasks. Results
  demonstrate that our method consistently achieves significant performance
  improvements while requiring less system cost compared to existing methods.
paper_ai_summary: A method for merging Large Language Models leverages layer-wise averaging and task-level expert routing to improve performance and reduce system cost.
paper_ai_keywords: Model merging Large Language Models LLMs parameter conflicts model routing layer-wise averaging task-level expert routing task arithmetic sparsity dense expert sparse experts out-of-distribution samples task uncertainty LLaMA Qwen