CMoE: Fast Carving of Mixture-of-Experts for Efficient LLM Inference
Large language models (LLMs) achieve impressive performance by scaling model
parameters, but this comes with significant inference overhead. Feed-forward
networks (FFNs), which dominate LLM parameters, exhibit high activation
sparsity in hidden neurons. To exploit this, researchers have proposed using a
mixture-of-experts (MoE) architecture, where only a subset of parameters is
activated. However, existing approaches often require extensive training data
and resources, limiting their practicality. We propose CMoE (Carved MoE), a
novel framework to efficiently carve MoE models from dense models. CMoE
achieves remarkable performance through efficient expert grouping and
lightweight adaptation. First, neurons are grouped into shared and routed
experts based on activation rates. Next, we construct a routing mechanism
without training from scratch, incorporating a differentiable routing process
and load balancing. Using modest data, CMoE produces a well-designed, usable
MoE from a 7B dense model within five minutes. With lightweight fine-tuning, it
achieves high-performance recovery in under an hour. We make our code publicly
available at https://github.com/JarvisPei/CMoE.
67a970920d2e1d1311d04053
Zehua Pei
claimed_verified
2025-02-10T09:50:03.881Z
6527c063e86758eb6ca800a1
/avatars/9091be87eea518209c1de9eebfa663c0.svg
JarvisPei
user
Eleven-P
67a970920d2e1d1311d04054
Lancheng Zou
claimed_verified
2025-02-10T13:12:04.847Z
65392c3429f8a911550fb9d8
/avatars/f12cd0ace82072817baea0d72f158de5.svg
LANCHENG ZOU
user
culczou
67a970920d2e1d1311d04055
Hui-Ling Zhen
67a970920d2e1d1311d04056
Xianzhi Yu
67a970920d2e1d1311d04057
Wulong Liu
admin_assigned
2025-02-10T16:18:49.935Z
67444dc518f0b9f39c48aa2f
/avatars/c911660165cc4c0abf6e0dcf6fa46034.svg
liuwulong
user
long202005589
67a970920d2e1d1311d04058
Sinno Jialin Pan
admin_assigned
2025-02-10T16:19:06.682Z
6751cc1807c0a99c402af739
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GPK4RlHBRFxbgrSKfkxUz.png
Sinno Pan
user
SinnoPan
67a970920d2e1d1311d04059
Mingxuan Yuan
67a970920d2e1d1311d0405a
Bei Yu
CMoE: Fast Carving of Mixture-of-Experts for Efficient LLM Inference
Large language models (LLMs) achieve impressive performance by scaling model
parameters, but this comes with significant inference overhead. Feed-forward
networks (FFNs), which dominate LLM parameters, exhibit high activation
sparsity in hidden neurons. To exploit this, researchers have proposed using a
mixture-of-experts (MoE) architecture, where only a subset of parameters is
activated. However, existing approaches often require extensive training data
and resources, limiting their practicality. We propose CMoE (Carved MoE), a
novel framework to efficiently carve MoE models from dense models. CMoE
achieves remarkable performance through efficient expert grouping and
lightweight adaptation. First, neurons are grouped into shared and routed
experts based on activation rates. Next, we construct a routing mechanism
without training from scratch, incorporating a differentiable routing process
and load balancing. Using modest data, CMoE produces a well-designed, usable
MoE from a 7B dense model within five minutes. With lightweight fine-tuning, it
achieves high-performance recovery in under an hour. We make our code publicly
available at https://github.com/JarvisPei/CMoE.
A novel framework, CMoE, efficiently converts dense large language models to MoE models using expert grouping and differentiable routing, achieving high performance with minimal resources.
large language models (LLMs) feed-forward networks (FFNs) activation sparsity mixture-of-experts (MoE) Carved MoE (CMoE) shared and routed experts differentiable routing load balancing