MoE-Mamba: Efficient Selective State Space Models with Mixture of
  Experts
State Space Models (SSMs) have become serious contenders in the field of
sequential modeling, challenging the dominance of Transformers. At the same
time, Mixture of Experts (MoE) has significantly improved Transformer-based
LLMs, including recent state-of-the-art open-source models. We propose that to
unlock the potential of SSMs for scaling, they should be combined with MoE. We
showcase this on Mamba, a recent SSM-based model that achieves remarkable,
Transformer-like performance. Our model, MoE-Mamba, outperforms both Mamba and
Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba
in 2.2x less training steps while preserving the inference performance gains of
Mamba against the Transformer.
659ccb1818ad5521982c6032
Maciej Pióro
admin_assigned
2024-01-09T08:15:29.785Z
643e7c8b3870da7f0ef20f36
/avatars/71291f75e2ab20818537159179f904a6.svg
Maciej Pióro
user
maciek-pioro
659ccb1818ad5521982c6033
Kamil Ciebiera
659ccb1818ad5521982c6034
Krystian Król
659ccb1818ad5521982c6035
Jan Ludziejewski
659ccb1818ad5521982c6036
Sebastian Jaszczur
MoE-Mamba: Efficient Selective State Space Models with Mixture of
  Experts
State Space Models (SSMs) have become serious contenders in the field of
sequential modeling, challenging the dominance of Transformers. At the same
time, Mixture of Experts (MoE) has significantly improved Transformer-based
LLMs, including recent state-of-the-art open-source models. We propose that to
unlock the potential of SSMs for scaling, they should be combined with MoE. We
showcase this on Mamba, a recent SSM-based model that achieves remarkable,
Transformer-like performance. Our model, MoE-Mamba, outperforms both Mamba and
Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba
in 2.2x less training steps while preserving the inference performance gains of
Mamba against the Transformer.
A combination of Mixture of Experts (MoE) with State Space Models (SSMs) improves model efficiency and performance compared to both standalone SSMs and MoE-enhanced Transformers.
State Space Models Sequential modeling Transformers Mixture of Experts MoE MoE-Mamba