title: CMoE: Fast Carving of Mixture-of-Experts for Efficient LLM Inference
summary: Large language models (LLMs) achieve impressive performance by scaling model
  parameters, but this comes with significant inference overhead. Feed-forward
  networks (FFNs), which dominate LLM parameters, exhibit high activation
  sparsity in hidden neurons. To exploit this, researchers have proposed using a
  mixture-of-experts (MoE) architecture, where only a subset of parameters is
  activated. However, existing approaches often require extensive training data
  and resources, limiting their practicality. We propose CMoE (Carved MoE), a
  novel framework to efficiently carve MoE models from dense models. CMoE
  achieves remarkable performance through efficient expert grouping and
  lightweight adaptation. First, neurons are grouped into shared and routed
  experts based on activation rates. Next, we construct a routing mechanism
  without training from scratch, incorporating a differentiable routing process
  and load balancing. Using modest data, CMoE produces a well-designed, usable
  MoE from a 7B dense model within five minutes. With lightweight fine-tuning, it
  achieves high-performance recovery in under an hour. We make our code publicly
  available at https://github.com/JarvisPei/CMoE.
paper_authors._id: 67a970920d2e1d1311d04053
paper_authors.name: Zehua Pei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:50:03.881Z
paper_authors._id: 6527c063e86758eb6ca800a1
paper_authors.avatarUrl: /avatars/9091be87eea518209c1de9eebfa663c0.svg
paper_authors.fullname: JarvisPei
paper_authors.type: user
paper_authors.user: Eleven-P
paper_authors._id: 67a970920d2e1d1311d04054
paper_authors.name: Lancheng Zou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T13:12:04.847Z
paper_authors._id: 65392c3429f8a911550fb9d8
paper_authors.avatarUrl: /avatars/f12cd0ace82072817baea0d72f158de5.svg
paper_authors.fullname: LANCHENG ZOU
paper_authors.type: user
paper_authors.user: culczou
paper_authors._id: 67a970920d2e1d1311d04055
paper_authors.name: Hui-Ling Zhen
paper_authors._id: 67a970920d2e1d1311d04056
paper_authors.name: Xianzhi Yu
paper_authors._id: 67a970920d2e1d1311d04057
paper_authors.name: Wulong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:18:49.935Z
paper_authors._id: 67444dc518f0b9f39c48aa2f
paper_authors.avatarUrl: /avatars/c911660165cc4c0abf6e0dcf6fa46034.svg
paper_authors.fullname: liuwulong
paper_authors.type: user
paper_authors.user: long202005589
paper_authors._id: 67a970920d2e1d1311d04058
paper_authors.name: Sinno Jialin Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:19:06.682Z
paper_authors._id: 6751cc1807c0a99c402af739
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GPK4RlHBRFxbgrSKfkxUz.png
paper_authors.fullname: Sinno Pan
paper_authors.type: user
paper_authors.user: SinnoPan
paper_authors._id: 67a970920d2e1d1311d04059
paper_authors.name: Mingxuan Yuan
paper_authors._id: 67a970920d2e1d1311d0405a
paper_authors.name: Bei Yu
paper_title: CMoE: Fast Carving of Mixture-of-Experts for Efficient LLM Inference
paper_summary: Large language models (LLMs) achieve impressive performance by scaling model
  parameters, but this comes with significant inference overhead. Feed-forward
  networks (FFNs), which dominate LLM parameters, exhibit high activation
  sparsity in hidden neurons. To exploit this, researchers have proposed using a
  mixture-of-experts (MoE) architecture, where only a subset of parameters is
  activated. However, existing approaches often require extensive training data
  and resources, limiting their practicality. We propose CMoE (Carved MoE), a
  novel framework to efficiently carve MoE models from dense models. CMoE
  achieves remarkable performance through efficient expert grouping and
  lightweight adaptation. First, neurons are grouped into shared and routed
  experts based on activation rates. Next, we construct a routing mechanism
  without training from scratch, incorporating a differentiable routing process
  and load balancing. Using modest data, CMoE produces a well-designed, usable
  MoE from a 7B dense model within five minutes. With lightweight fine-tuning, it
  achieves high-performance recovery in under an hour. We make our code publicly
  available at https://github.com/JarvisPei/CMoE.
paper_ai_summary: A novel framework, CMoE, efficiently converts dense large language models to MoE models using expert grouping and differentiable routing, achieving high performance with minimal resources.
paper_ai_keywords: large language models (LLMs) feed-forward networks (FFNs) activation sparsity mixture-of-experts (MoE) Carved MoE (CMoE) shared and routed experts differentiable routing load balancing