title: Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
summary: Linear Sequence Modeling (LSM) like linear attention, state space models and
  linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant
  architectural improvements. In this paper, we introduce Linear-MoE, a
  production-level system for modeling and training large-scale models that
  integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules
  for linear-complexity sequence modeling and MoE layers for sparsely activation,
  aiming to offer high performance with efficient training. The Linear-MoE system
  comprises: 1) Modeling subsystem, which provides a unified framework supporting
  all instances of LSM. and 2) Training subsystem, which facilitates efficient
  training by incorporating various advanced parallelism technologies,
  particularly Sequence Parallelism designed for Linear-MoE models. Additionally,
  we explore hybrid models that combine Linear-MoE layers with standard
  Transformer-MoE layers with its Sequence Parallelism to further enhance model
  flexibility and performance. Evaluations on two model series, A0.3B-2B and
  A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining
  competitive performance on various benchmarks, showcasing its potential as a
  next-generation foundational model architecture. Code:
  https://github.com/OpenSparseLLMs/Linear-MoE.
paper_authors._id: 67ceab4132a6585cecad2c36
paper_authors.name: Weigao Sun
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-10T09:09:22.436Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 67ceab4132a6585cecad2c37
paper_authors.name: Disen Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T10:13:18.306Z
paper_authors._id: 66ea643899af9ac3463639b1
paper_authors.avatarUrl: /avatars/252d470e761a57834dee3dbc60dfefed.svg
paper_authors.fullname: Disen Lan
paper_authors.type: user
paper_authors.user: landisen
paper_authors._id: 67ceab4132a6585cecad2c38
paper_authors.name: Tong Zhu
paper_authors._id: 67ceab4132a6585cecad2c39
paper_authors.name: Xiaoye Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:19:56.928Z
paper_authors._id: 64cb54da1af278541d663708
paper_authors.avatarUrl: /avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
paper_authors.fullname: Xiaoye Qu
paper_authors.type: user
paper_authors.user: Xiaoye08
paper_authors._id: 67ceab4132a6585cecad2c3a
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:52:46.691Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_title: Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
paper_summary: Linear Sequence Modeling (LSM) like linear attention, state space models and
  linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant
  architectural improvements. In this paper, we introduce Linear-MoE, a
  production-level system for modeling and training large-scale models that
  integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules
  for linear-complexity sequence modeling and MoE layers for sparsely activation,
  aiming to offer high performance with efficient training. The Linear-MoE system
  comprises: 1) Modeling subsystem, which provides a unified framework supporting
  all instances of LSM. and 2) Training subsystem, which facilitates efficient
  training by incorporating various advanced parallelism technologies,
  particularly Sequence Parallelism designed for Linear-MoE models. Additionally,
  we explore hybrid models that combine Linear-MoE layers with standard
  Transformer-MoE layers with its Sequence Parallelism to further enhance model
  flexibility and performance. Evaluations on two model series, A0.3B-2B and
  A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining
  competitive performance on various benchmarks, showcasing its potential as a
  next-generation foundational model architecture. Code:
  https://github.com/OpenSparseLLMs/Linear-MoE.
paper_ai_summary: Linear-MoE, integrating linear sequence modeling with Mixture-of-Experts, achieves efficient training and competitive performance using advanced parallelism like Sequence Parallelism.
paper_ai_keywords: linear attention state space models linear RNNs Mixture-of-Experts Linear-MoE Linear Sequence Modeling Sequence Parallelism Transformer-MoE Sequence Parallelism foundational model architecture