title: PreMoe: Lightening MoEs on Constrained Memory by Expert Pruning and
    Retrieval
summary: Mixture-of-experts (MoE) architectures enable scaling large language models
  (LLMs) to vast parameter counts without a proportional rise in computational
  costs. However, the significant memory demands of large MoE models hinder their
  deployment across various computational environments, from cloud servers to
  consumer devices. This study first demonstrates pronounced task-specific
  specialization in expert activation patterns within MoE layers. Building on
  this, we introduce PreMoe, a novel framework that enables efficient deployment
  of massive MoE models in memory-constrained environments. PreMoe features two
  main components: probabilistic expert pruning (PEP) and task-adaptive expert
  retrieval (TAER). PEP employs a new metric, the task-conditioned expected
  selection score (TCESS), derived from router logits to quantify expert
  importance for specific tasks, thereby identifying a minimal set of critical
  experts. TAER leverages these task-specific expert importance profiles for
  efficient inference. It pre-computes and stores compact expert patterns for
  diverse tasks. When a user query is received, TAER rapidly identifies the most
  relevant stored task pattern and reconstructs the model by loading only the
  small subset of experts crucial for that task. This approach dramatically
  reduces the memory footprint across all deployment scenarios. DeepSeek-R1 671B
  maintains 97.2\% accuracy on MATH500 when pruned to 8/128 configuration (50\%
  expert reduction), and still achieves 72.0\% with aggressive 8/32 pruning
  (87.5\% expert reduction). Pangu-Ultra-MoE 718B achieves 97.15\% on MATH500 and
  81.3\% on AIME24 with 8/128 pruning, while even more aggressive pruning to 4/64
  (390GB memory) preserves 96.95\% accuracy on MATH500. We make our code publicly
  available at https://github.com/JarvisPei/PreMoe.
paper_authors._id: 683733543eb396dd7b9c8287
paper_authors.name: Zehua Pei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:13:42.952Z
paper_authors._id: 6527c063e86758eb6ca800a1
paper_authors.avatarUrl: /avatars/9091be87eea518209c1de9eebfa663c0.svg
paper_authors.fullname: JarvisPei
paper_authors.type: user
paper_authors.user: Eleven-P
paper_authors._id: 683733543eb396dd7b9c8288
paper_authors.name: Ying Zhang
paper_authors._id: 683733543eb396dd7b9c8289
paper_authors.name: Hui-Ling Zhen
paper_authors._id: 683733543eb396dd7b9c828a
paper_authors.name: Xianzhi Yu
paper_authors._id: 683733543eb396dd7b9c828b
paper_authors.name: Wulong Liu
paper_authors._id: 683733543eb396dd7b9c828c
paper_authors.name: Sinno Jialin Pan
paper_authors._id: 683733543eb396dd7b9c828d
paper_authors.name: Mingxuan Yuan
paper_authors._id: 683733543eb396dd7b9c828e
paper_authors.name: Bei Yu
paper_title: PreMoe: Lightening MoEs on Constrained Memory by Expert Pruning and
    Retrieval
paper_summary: Mixture-of-experts (MoE) architectures enable scaling large language models
  (LLMs) to vast parameter counts without a proportional rise in computational
  costs. However, the significant memory demands of large MoE models hinder their
  deployment across various computational environments, from cloud servers to
  consumer devices. This study first demonstrates pronounced task-specific
  specialization in expert activation patterns within MoE layers. Building on
  this, we introduce PreMoe, a novel framework that enables efficient deployment
  of massive MoE models in memory-constrained environments. PreMoe features two
  main components: probabilistic expert pruning (PEP) and task-adaptive expert
  retrieval (TAER). PEP employs a new metric, the task-conditioned expected
  selection score (TCESS), derived from router logits to quantify expert
  importance for specific tasks, thereby identifying a minimal set of critical
  experts. TAER leverages these task-specific expert importance profiles for
  efficient inference. It pre-computes and stores compact expert patterns for
  diverse tasks. When a user query is received, TAER rapidly identifies the most
  relevant stored task pattern and reconstructs the model by loading only the
  small subset of experts crucial for that task. This approach dramatically
  reduces the memory footprint across all deployment scenarios. DeepSeek-R1 671B
  maintains 97.2\% accuracy on MATH500 when pruned to 8/128 configuration (50\%
  expert reduction), and still achieves 72.0\% with aggressive 8/32 pruning
  (87.5\% expert reduction). Pangu-Ultra-MoE 718B achieves 97.15\% on MATH500 and
  81.3\% on AIME24 with 8/128 pruning, while even more aggressive pruning to 4/64
  (390GB memory) preserves 96.95\% accuracy on MATH500. We make our code publicly
  available at https://github.com/JarvisPei/PreMoe.
paper_ai_summary: PreMoe framework enables efficient deployment of large MoE language models in memory-constrained environments by pruning and retrieving task-specific experts.
paper_ai_keywords: mixture-of-experts (MoE) large language models (LLMs) probabilistic expert pruning (PEP) task-adaptive expert retrieval (TAER) task-conditioned expected selection score (TCESS) DeepSeek-R1 Pangu-Ultra-MoE MATH500 AIME24