title: MoE-Mamba: Efficient Selective State Space Models with Mixture of
    Experts
summary: State Space Models (SSMs) have become serious contenders in the field of
  sequential modeling, challenging the dominance of Transformers. At the same
  time, Mixture of Experts (MoE) has significantly improved Transformer-based
  LLMs, including recent state-of-the-art open-source models. We propose that to
  unlock the potential of SSMs for scaling, they should be combined with MoE. We
  showcase this on Mamba, a recent SSM-based model that achieves remarkable,
  Transformer-like performance. Our model, MoE-Mamba, outperforms both Mamba and
  Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba
  in 2.2x less training steps while preserving the inference performance gains of
  Mamba against the Transformer.
paper_authors._id: 659ccb1818ad5521982c6032
paper_authors.name: Maciej Pióro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:15:29.785Z
paper_authors._id: 643e7c8b3870da7f0ef20f36
paper_authors.avatarUrl: /avatars/71291f75e2ab20818537159179f904a6.svg
paper_authors.fullname: Maciej Pióro
paper_authors.type: user
paper_authors.user: maciek-pioro
paper_authors._id: 659ccb1818ad5521982c6033
paper_authors.name: Kamil Ciebiera
paper_authors._id: 659ccb1818ad5521982c6034
paper_authors.name: Krystian Król
paper_authors._id: 659ccb1818ad5521982c6035
paper_authors.name: Jan Ludziejewski
paper_authors._id: 659ccb1818ad5521982c6036
paper_authors.name: Sebastian Jaszczur
paper_title: MoE-Mamba: Efficient Selective State Space Models with Mixture of
    Experts
paper_summary: State Space Models (SSMs) have become serious contenders in the field of
  sequential modeling, challenging the dominance of Transformers. At the same
  time, Mixture of Experts (MoE) has significantly improved Transformer-based
  LLMs, including recent state-of-the-art open-source models. We propose that to
  unlock the potential of SSMs for scaling, they should be combined with MoE. We
  showcase this on Mamba, a recent SSM-based model that achieves remarkable,
  Transformer-like performance. Our model, MoE-Mamba, outperforms both Mamba and
  Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba
  in 2.2x less training steps while preserving the inference performance gains of
  Mamba against the Transformer.
paper_ai_keywords: State Space Models (SSMs) Transformer Mixture of Experts (MoE) Large Language Models (LLMs) Mamba MoE-Mamba Tensor-Web Compiling Tensor-Web Compiling Co-Operating Mamba against the Transformer