title: Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with
    System Co-Design
summary: The proliferation of large language models (LLMs) has led to the adoption of
  Mixture-of-Experts (MoE) architectures that dynamically leverage specialized
  subnetworks for improved efficiency and performance. Despite their benefits,
  MoE models face significant challenges during inference, including inefficient
  memory management and suboptimal batching, due to misaligned design choices
  between the model architecture and the system policies. Furthermore, the
  conventional approach of training MoEs from scratch is increasingly prohibitive
  in terms of cost. In this paper, we propose a novel framework Read-ME that
  transforms pre-trained dense LLMs into smaller MoE models (in contrast to
  "upcycling" generalist MoEs), avoiding the high costs of ground-up training.
  Our approach employs activation sparsity to extract experts. To compose
  experts, we examine the widely-adopted layer-wise router design and show its
  redundancy, and thus we introduce the pre-gating router decoupled from the MoE
  backbone that facilitates system-friendly pre-computing and lookahead
  scheduling, enhancing expert-aware batching and caching. Our codesign therefore
  addresses critical gaps on both the algorithmic and system fronts, establishing
  a scalable and efficient alternative for LLM inference in resource-constrained
  settings. Read-ME outperforms other popular open-source dense models of similar
  scales, achieving improvements of up to 10.1% on MMLU, and improving mean
  end-to-end latency up to 6.1%. Codes are available at:
  https://github.com/VITA-Group/READ-ME.
paper_authors._id: 671ff0de1925c496303fa4de
paper_authors.name: Ruisi Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:34.650Z
paper_authors._id: 63c99e2ddfac8071d01b89da
paper_authors.avatarUrl: /avatars/196af8c8498aa7a930e77c0a169da678.svg
paper_authors.fullname: Ruisi Cai
paper_authors.type: user
paper_authors.user: CCCCRS
paper_authors._id: 671ff0de1925c496303fa4df
paper_authors.name: Yeonju Ro
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:32.989Z
paper_authors._id: 654a83fcdff2f49007ce439a
paper_authors.avatarUrl: /avatars/0ea261054b0c81a0aa3008837b1708e7.svg
paper_authors.fullname: Yeonju Ro
paper_authors.type: user
paper_authors.user: julia777ro
paper_authors._id: 671ff0de1925c496303fa4e0
paper_authors.name: Geon-Woo Kim
paper_authors._id: 671ff0de1925c496303fa4e1
paper_authors.name: Peihao Wang
paper_authors._id: 671ff0de1925c496303fa4e2
paper_authors.name: Babak Ehteshami Bejnordi
paper_authors._id: 671ff0de1925c496303fa4e3
paper_authors.name: Aditya Akella
paper_authors._id: 671ff0de1925c496303fa4e4
paper_authors.name: Zhangyang Wang
paper_title: Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with
    System Co-Design
paper_summary: The proliferation of large language models (LLMs) has led to the adoption of
  Mixture-of-Experts (MoE) architectures that dynamically leverage specialized
  subnetworks for improved efficiency and performance. Despite their benefits,
  MoE models face significant challenges during inference, including inefficient
  memory management and suboptimal batching, due to misaligned design choices
  between the model architecture and the system policies. Furthermore, the
  conventional approach of training MoEs from scratch is increasingly prohibitive
  in terms of cost. In this paper, we propose a novel framework Read-ME that
  transforms pre-trained dense LLMs into smaller MoE models (in contrast to
  "upcycling" generalist MoEs), avoiding the high costs of ground-up training.
  Our approach employs activation sparsity to extract experts. To compose
  experts, we examine the widely-adopted layer-wise router design and show its
  redundancy, and thus we introduce the pre-gating router decoupled from the MoE
  backbone that facilitates system-friendly pre-computing and lookahead
  scheduling, enhancing expert-aware batching and caching. Our codesign therefore
  addresses critical gaps on both the algorithmic and system fronts, establishing
  a scalable and efficient alternative for LLM inference in resource-constrained
  settings. Read-ME outperforms other popular open-source dense models of similar
  scales, achieving improvements of up to 10.1% on MMLU, and improving mean
  end-to-end latency up to 6.1%. Codes are available at:
  https://github.com/VITA-Group/READ-ME.
paper_ai_keywords: Mixture-of-Experts (MoE) architectures specialized subnetworks activation sparsity layer-wise router design pre-gating router MoE backbone pre-computing lookahead scheduling expert-aware batching caching Read-ME framework MMLU