title: BlackMamba: Mixture of Experts for State-Space Models
summary: State-space models (SSMs) have recently demonstrated competitive performance
  to transformers at large-scale language modeling benchmarks while achieving
  linear time and memory complexity as a function of sequence length. Mamba, a
  recently released SSM model, shows impressive performance in both language
  modeling and long sequence processing tasks. Simultaneously, mixture-of-expert
  (MoE) models have shown remarkable performance while significantly reducing the
  compute and latency costs of inference at the expense of a larger memory
  footprint. In this paper, we present BlackMamba, a novel architecture that
  combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate
  that BlackMamba performs competitively against both Mamba and transformer
  baselines, and outperforms in inference and training FLOPs. We fully train and
  open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a
  custom dataset. We show that BlackMamba inherits and combines both of the
  benefits of SSM and MoE architectures, combining linear-complexity generation
  from SSM with cheap and fast inference from MoE. We release all weights,
  checkpoints, and inference code open-source. Inference code at:
  https://github.com/Zyphra/BlackMamba
paper_authors._id: 65c1b21bca9755314bf8178c
paper_authors.name: Quentin Anthony
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-06T17:52:09.150Z
paper_authors._id: 65bc13717c6ad1994b6619e9
paper_authors.avatarUrl: /avatars/8e97e25e2de30f6bea9f9bc94418ff0f.svg
paper_authors.fullname: Quentin Anthony
paper_authors.type: user
paper_authors.user: qanthony-z
paper_authors._id: 65c1b21bca9755314bf8178d
paper_authors.name: Yury Tokpanov
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-06T04:14:19.754Z
paper_authors._id: 65455aca468722e935103b17
paper_authors.avatarUrl: /avatars/29786a7df6541aa1e5d408eb06b0bd96.svg
paper_authors.fullname: Yury Tokpanov
paper_authors.type: user
paper_authors.user: yury-zyphra
paper_authors._id: 65c1b21bca9755314bf8178e
paper_authors.name: Paolo Glorioso
paper_authors._id: 65c1b21bca9755314bf8178f
paper_authors.name: Beren Millidge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:41:27.149Z
paper_authors._id: 65c05e75c084467acab2f84a
paper_authors.avatarUrl: /avatars/255e98fe436d9d9252975e842755c0ff.svg
paper_authors.fullname: Beren Millidge
paper_authors.type: user
paper_authors.user: BerenMillidge
paper_title: BlackMamba: Mixture of Experts for State-Space Models
paper_summary: State-space models (SSMs) have recently demonstrated competitive performance
  to transformers at large-scale language modeling benchmarks while achieving
  linear time and memory complexity as a function of sequence length. Mamba, a
  recently released SSM model, shows impressive performance in both language
  modeling and long sequence processing tasks. Simultaneously, mixture-of-expert
  (MoE) models have shown remarkable performance while significantly reducing the
  compute and latency costs of inference at the expense of a larger memory
  footprint. In this paper, we present BlackMamba, a novel architecture that
  combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate
  that BlackMamba performs competitively against both Mamba and transformer
  baselines, and outperforms in inference and training FLOPs. We fully train and
  open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a
  custom dataset. We show that BlackMamba inherits and combines both of the
  benefits of SSM and MoE architectures, combining linear-complexity generation
  from SSM with cheap and fast inference from MoE. We release all weights,
  checkpoints, and inference code open-source. Inference code at:
  https://github.com/Zyphra/BlackMamba
paper_ai_keywords: state-space models (SSMs) transformers Mamba mixture-of-expert (MoE) models BlackMamba linear time and memory complexity long sequence processing tasks inference training FLOPs open-source custom dataset unified custom dataset linear-complexity generation cheap and fast inference state-space models (SSMs) mixture-of-expert (MoE) models BlackMamba linear time and memory complexity long sequence processing tasks inference FLOPs open-source linear-complexity generation cheap and fast inference