title: HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient
    MoE Inference
summary: The Mixture of Experts (MoE) architecture has demonstrated significant
  advantages as it enables to increase the model capacity without a proportional
  increase in computation. However, the large MoE model size still introduces
  substantial memory demands, which usually requires expert offloading on
  resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU
  inference has been proposed to leverage CPU computation to reduce expert
  loading overhead but faces major challenges: on one hand, the expert activation
  patterns of MoE models are highly unstable, rendering the fixed mapping
  strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU
  schedule for MoE is inherently complex due to the diverse expert sizes,
  structures, uneven workload distribution, etc. To address these challenges, in
  this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that
  improves resource utilization through a novel CPU-GPU scheduling and cache
  management system. HybriMoE introduces (i) a dynamic intra-layer scheduling
  strategy to balance workloads across CPU and GPU, (ii) an impact-driven
  inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to
  mitigate expert activation instability. We implement HybriMoE on top of the
  kTransformers framework and evaluate it on three widely used MoE-based LLMs.
  Experimental results demonstrate that HybriMoE achieves an average speedup of
  1.33times in the prefill stage and 1.70times in the decode stage compared
  to state-of-the-art hybrid MoE inference framework. Our code is available at:
  https://github.com/PKU-SEC-Lab/HybriMoE.
paper_authors._id: 67f660097c5b9743c24119f5
paper_authors.name: Shuzhang Zhong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:35.851Z
paper_authors._id: 64cb667f00377e2848753e33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64cb667f00377e2848753e33/rCGn1p0YLIe6HA_ft0z9L.png
paper_authors.fullname: kevin zhong
paper_authors.type: user
paper_authors.user: kevinzhong
paper_authors._id: 67f660097c5b9743c24119f6
paper_authors.name: Yanfan Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:55:46.835Z
paper_authors._id: 646de8fb4ad7f907279ed6b3
paper_authors.avatarUrl: /avatars/d38f55fe0676929182d376f6e8af8cab.svg
paper_authors.fullname: Sa lan
paper_authors.type: user
paper_authors.user: YanfanSun
paper_authors._id: 67f660097c5b9743c24119f7
paper_authors.name: Ling Liang
paper_authors._id: 67f660097c5b9743c24119f8
paper_authors.name: Runsheng Wang
paper_authors._id: 67f660097c5b9743c24119f9
paper_authors.name: Ru Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:49:21.494Z
paper_authors._id: 652639bfd5a67051c6692161
paper_authors.avatarUrl: /avatars/12cff4d93ab4e09b6a9b72f2a44a3b1a.svg
paper_authors.fullname: ruhuang
paper_authors.type: user
paper_authors.user: ruhuang
paper_authors._id: 67f660097c5b9743c24119fa
paper_authors.name: Meng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T07:57:57.547Z
paper_authors._id: 65211f76c709aaca9a8bc5c7
paper_authors.avatarUrl: /avatars/4fa7619cb18b7ad6869e8fe1547c02cf.svg
paper_authors.fullname: Meng Li
paper_authors.type: user
paper_authors.user: ml-pku
paper_title: HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient
    MoE Inference
paper_summary: The Mixture of Experts (MoE) architecture has demonstrated significant
  advantages as it enables to increase the model capacity without a proportional
  increase in computation. However, the large MoE model size still introduces
  substantial memory demands, which usually requires expert offloading on
  resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU
  inference has been proposed to leverage CPU computation to reduce expert
  loading overhead but faces major challenges: on one hand, the expert activation
  patterns of MoE models are highly unstable, rendering the fixed mapping
  strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU
  schedule for MoE is inherently complex due to the diverse expert sizes,
  structures, uneven workload distribution, etc. To address these challenges, in
  this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that
  improves resource utilization through a novel CPU-GPU scheduling and cache
  management system. HybriMoE introduces (i) a dynamic intra-layer scheduling
  strategy to balance workloads across CPU and GPU, (ii) an impact-driven
  inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to
  mitigate expert activation instability. We implement HybriMoE on top of the
  kTransformers framework and evaluate it on three widely used MoE-based LLMs.
  Experimental results demonstrate that HybriMoE achieves an average speedup of
  1.33times in the prefill stage and 1.70times in the decode stage compared
  to state-of-the-art hybrid MoE inference framework. Our code is available at:
  https://github.com/PKU-SEC-Lab/HybriMoE.
paper_ai_summary: HybriMoE enhances the hybrid CPU-GPU inference of MoE models through dynamic scheduling, prefetching, and caching to improve resource utilization and reduce expert activation instability.
paper_ai_keywords: mixture of experts (MoE) hybrid CPU-GPU inference expert activation patterns dynamic intra-layer scheduling impact-driven inter-layer prefetching score-based caching kTransformers large language models (LLMs) speedup