SlimMoE: Structured Compression of Large MoE Models via Expert Slimming
  and Distillation
The Mixture of Experts (MoE) architecture has emerged as a powerful paradigm
for scaling large language models (LLMs) while maintaining inference
efficiency. However, their enormous memory requirements make them prohibitively
expensive to fine-tune or deploy in resource-constrained environments. To
address this challenge, we introduce SlimMoE, a multi-stage compression
framework for transforming large MoE models into much smaller, efficient
variants without incurring the prohibitive costs of training from scratch. Our
method systematically reduces parameter counts by slimming experts and
transferring knowledge through intermediate stages, effectively mitigating the
performance degradation common in one-shot pruning approaches. Using this
framework, we compress Phi 3.5-MoE (41.9B total/6.6B activated parameters) to
create Phi-mini-MoE (7.6B total/2.4B activated parameters) and Phi-tiny-MoE
(3.8B total/1.1B activated parameters) using only 400B tokens--less than 10% of
the original model's training data. These compressed models can be fine-tuned
on a single GPU (A100 for Phi-mini-MoE, A6000 for Phi-tiny-MoE), making them
highly suitable for academic and resource-limited settings. Our experiments
demonstrate that these compressed models outperform others of similar size and
remain competitive with larger models. For instance, Phi-mini-MoE achieves
similar or better performance to Phi-3-mini using only 2/3 of the activated
parameters and yields comparable MMLU scores to Llama 3.1 8B despite having
significantly lower latency. Our findings demonstrate that structured pruning
combined with staged distillation offers an effective path to creating
high-quality, compact MoE models, paving the way for broader adoption of MoE
architectures. We make our models publicly available at
https://huggingface.co/microsoft/Phi-mini-MoE-instruct and
https://huggingface.co/microsoft/Phi-tiny-MoE-instruct .
685a02460e4ad7e2197584a9
Zichong Li
claimed_verified
2025-06-24T08:08:56.270Z
659c6a50615d5e661222fe16
/avatars/8a946482e49a821dbe397dc3898f22c5.svg
Zichong Li
user
Pearush
685a02460e4ad7e2197584aa
Chen Liang
claimed_verified
2025-07-13T07:29:59.450Z
63e6b5e22d2c508de9001afd
/avatars/43cec9e8b8d490bd259e383954846a1e.svg
Chen Liang
user
cliang1453
685a02460e4ad7e2197584ab
Zixuan Zhang
685a02460e4ad7e2197584ac
Ilgee Hong
685a02460e4ad7e2197584ad
Young Jin Kim
claimed_verified
2025-07-05T07:54:05.501Z
63f5173bb51da4d61da6c038
/avatars/0ee530cf80476aa3985c4d591cd384a1.svg
Young Jin Kim
user
ykim362
685a02460e4ad7e2197584ae
Weizhu Chen
685a02460e4ad7e2197584af
Tuo Zhao
SlimMoE: Structured Compression of Large MoE Models via Expert Slimming
  and Distillation
The Mixture of Experts (MoE) architecture has emerged as a powerful paradigm
for scaling large language models (LLMs) while maintaining inference
efficiency. However, their enormous memory requirements make them prohibitively
expensive to fine-tune or deploy in resource-constrained environments. To
address this challenge, we introduce SlimMoE, a multi-stage compression
framework for transforming large MoE models into much smaller, efficient
variants without incurring the prohibitive costs of training from scratch. Our
method systematically reduces parameter counts by slimming experts and
transferring knowledge through intermediate stages, effectively mitigating the
performance degradation common in one-shot pruning approaches. Using this
framework, we compress Phi 3.5-MoE (41.9B total/6.6B activated parameters) to
create Phi-mini-MoE (7.6B total/2.4B activated parameters) and Phi-tiny-MoE
(3.8B total/1.1B activated parameters) using only 400B tokens--less than 10% of
the original model's training data. These compressed models can be fine-tuned
on a single GPU (A100 for Phi-mini-MoE, A6000 for Phi-tiny-MoE), making them
highly suitable for academic and resource-limited settings. Our experiments
demonstrate that these compressed models outperform others of similar size and
remain competitive with larger models. For instance, Phi-mini-MoE achieves
similar or better performance to Phi-3-mini using only 2/3 of the activated
parameters and yields comparable MMLU scores to Llama 3.1 8B despite having
significantly lower latency. Our findings demonstrate that structured pruning
combined with staged distillation offers an effective path to creating
high-quality, compact MoE models, paving the way for broader adoption of MoE
architectures. We make our models publicly available at
https://huggingface.co/microsoft/Phi-mini-MoE-instruct and
https://huggingface.co/microsoft/Phi-tiny-MoE-instruct .
SlimMoE compresses large MoE models into smaller, efficient variants using multi-stage compression without full retraining, maintaining competitive performance with significantly fewer resources.
Mixture of Experts (MoE) large language models (LLMs) parameter counts knowledge transfer one-shot pruning Phi 3.5-MoE Phi-mini-MoE Phi-tiny-MoE structured pruning staged distillation MMLU scores latency