title: Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for
    Sparse Architectural Large Language Models
summary: Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
  Language Models (LLMs) with constrained resources. Although there have been
  various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
  LLMs is still underexplored. In this work, we study the PEFT method for LLMs
  with the Mixture-of-Experts (MoE) architecture and the contents of this work
  are mainly threefold: (1) We investigate the dispersion degree of the activated
  experts in customized tasks, and found that the routing distribution for a
  specific task tends to be highly concentrated, while the distribution of
  activated experts varies significantly across different tasks. (2) We propose
  Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
  to downstream tasks while freezing the other experts and modules; experimental
  results demonstrate that our method not only improves the tuning efficiency,
  but also matches or even surpasses the performance of full-parameter
  fine-tuning. (3) We further analyze the impact of the MoE architecture on
  expert-specialized fine-tuning. We find that MoE models with finer-grained
  experts are more advantageous in selecting the combination of experts that are
  most relevant to downstream tasks, thereby enhancing both the training
  efficiency and effectiveness.
paper_authors._id: 6686701e443833e90fc83875
paper_authors.name: Zihan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:41:08.760Z
paper_authors._id: 63318e014bd187e9ce05cec6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63318e014bd187e9ce05cec6/rKN84BAXF_u0osSu6KJy5.jpeg
paper_authors.fullname: Zihan Wang
paper_authors.type: user
paper_authors.user: ZihanWang314
paper_authors._id: 6686701e443833e90fc83876
paper_authors.name: Deli Chen
paper_authors._id: 6686701e443833e90fc83877
paper_authors.name: Damai Dai
paper_authors._id: 6686701e443833e90fc83878
paper_authors.name: Runxin Xu
paper_authors._id: 6686701e443833e90fc83879
paper_authors.name: Zhuoshu Li
paper_authors._id: 6686701e443833e90fc8387a
paper_authors.name: Y. Wu
paper_title: Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for
    Sparse Architectural Large Language Models
paper_summary: Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
  Language Models (LLMs) with constrained resources. Although there have been
  various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
  LLMs is still underexplored. In this work, we study the PEFT method for LLMs
  with the Mixture-of-Experts (MoE) architecture and the contents of this work
  are mainly threefold: (1) We investigate the dispersion degree of the activated
  experts in customized tasks, and found that the routing distribution for a
  specific task tends to be highly concentrated, while the distribution of
  activated experts varies significantly across different tasks. (2) We propose
  Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
  to downstream tasks while freezing the other experts and modules; experimental
  results demonstrate that our method not only improves the tuning efficiency,
  but also matches or even surpasses the performance of full-parameter
  fine-tuning. (3) We further analyze the impact of the MoE architecture on
  expert-specialized fine-tuning. We find that MoE models with finer-grained
  experts are more advantageous in selecting the combination of experts that are
  most relevant to downstream tasks, thereby enhancing both the training
  efficiency and effectiveness.
paper_ai_summary: Research on parameter-efficient fine-tuning for Large Language Models with Mixture-of-Experts architecture reveals that task-specific expert tuning can enhance performance and efficiency.
paper_ai_keywords: Parameter-efficient fine-tuning PEFT Large Language Models LLMs Mixture-of-Experts MoE dispersion degree activated experts routing distribution Expert-Specialized Fine-Tuning ESFT full-parameter fine-tuning training efficiency effectiveness