title: Not All Models Suit Expert Offloading: On Local Routing Consistency of
    Mixture-of-Expert Models
summary: Mixture-of-Experts (MoE) enables efficient scaling of large language models
  (LLMs) with sparsely activated experts during inference. To effectively deploy
  large MoE models on memory-constrained devices, many systems introduce *expert
  offloading* that caches a subset of experts in fast memory, leaving others on
  slow memory to run on CPU or load on demand. While some research has exploited
  the locality of expert activations, where consecutive tokens activate similar
  experts, the degree of this **local routing consistency** varies across models
  and remains understudied. In this paper, we propose two metrics to measure
  local routing consistency of MoE models: (1) **Segment Routing Best Performance
  (SRP)**, which evaluates how well a fixed group of experts can cover the needs
  of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which
  measures the optimal segment-level cache hit rate under a given cache size
  limit. We analyzed 20 MoE LLMs with diverse sizes and architectures and found
  that models that apply MoE on every layer and do not use shared experts exhibit
  the highest local routing consistency. We further showed that
  domain-specialized experts contribute more to routing consistency than
  vocabulary-specialized ones, and that most models can balance between cache
  effectiveness and efficiency with cache sizes approximately 2x the active
  experts. These findings pave the way for memory-efficient MoE design and
  deployment without compromising inference speed. We publish the code for
  replicating experiments at https://github.com/ljcleo/moe-lrc .
paper_authors._id: 6830894db51948863e05b68c
paper_authors.name: Jingcong Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:13:21.361Z
paper_authors._id: 64bfa1401d40292dd32f93d7
paper_authors.avatarUrl: /avatars/39a3d1772e0bc54f9bb2db7ce7047784.svg
paper_authors.fullname: Leo Liang
paper_authors.type: user
paper_authors.user: ljcleo
paper_authors._id: 6830894db51948863e05b68d
paper_authors.name: Siyuan Wang
paper_authors._id: 6830894db51948863e05b68e
paper_authors.name: Miren Tian
paper_authors._id: 6830894db51948863e05b68f
paper_authors.name: Yitong Li
paper_authors._id: 6830894db51948863e05b690
paper_authors.name: Duyu Tang
paper_authors._id: 6830894db51948863e05b691
paper_authors.name: Zhongyu Wei
paper_title: Not All Models Suit Expert Offloading: On Local Routing Consistency of
    Mixture-of-Expert Models
paper_summary: Mixture-of-Experts (MoE) enables efficient scaling of large language models
  (LLMs) with sparsely activated experts during inference. To effectively deploy
  large MoE models on memory-constrained devices, many systems introduce *expert
  offloading* that caches a subset of experts in fast memory, leaving others on
  slow memory to run on CPU or load on demand. While some research has exploited
  the locality of expert activations, where consecutive tokens activate similar
  experts, the degree of this **local routing consistency** varies across models
  and remains understudied. In this paper, we propose two metrics to measure
  local routing consistency of MoE models: (1) **Segment Routing Best Performance
  (SRP)**, which evaluates how well a fixed group of experts can cover the needs
  of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which
  measures the optimal segment-level cache hit rate under a given cache size
  limit. We analyzed 20 MoE LLMs with diverse sizes and architectures and found
  that models that apply MoE on every layer and do not use shared experts exhibit
  the highest local routing consistency. We further showed that
  domain-specialized experts contribute more to routing consistency than
  vocabulary-specialized ones, and that most models can balance between cache
  effectiveness and efficiency with cache sizes approximately 2x the active
  experts. These findings pave the way for memory-efficient MoE design and
  deployment without compromising inference speed. We publish the code for
  replicating experiments at https://github.com/ljcleo/moe-lrc .
paper_ai_summary: MoE models achieve efficient scaling in LLMs with expert offloading, emphasizing the importance of local routing consistency and cache effectiveness.
paper_ai_keywords: Mixture-of-Experts (MoE) large language models (LLMs) expert offloading fast memory slow memory local routing consistency Segment Routing Best Performance (SRP) Segment Cache Best Hit Rate (SCH) domain-specialized experts vocabulary-specialized experts