title: Mamba: Linear-Time Sequence Modeling with Selective State Spaces
summary: Foundation models, now powering most of the exciting applications in deep
  learning, are almost universally based on the Transformer architecture and its
  core attention module. Many subquadratic-time architectures such as linear
  attention, gated convolution and recurrent models, and structured state space
  models (SSMs) have been developed to address Transformers' computational
  inefficiency on long sequences, but they have not performed as well as
  attention on important modalities such as language. We identify that a key
  weakness of such models is their inability to perform content-based reasoning,
  and make several improvements. First, simply letting the SSM parameters be
  functions of the input addresses their weakness with discrete modalities,
  allowing the model to selectively propagate or forget information along the
  sequence length dimension depending on the current token. Second, even though
  this change prevents the use of efficient convolutions, we design a
  hardware-aware parallel algorithm in recurrent mode. We integrate these
  selective SSMs into a simplified end-to-end neural network architecture without
  attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5times
  higher throughput than Transformers) and linear scaling in sequence length, and
  its performance improves on real data up to million-length sequences. As a
  general sequence model backbone, Mamba achieves state-of-the-art performance
  across several modalities such as language, audio, and genomics. On language
  modeling, our Mamba-3B model outperforms Transformers of the same size and
  matches Transformers twice its size, both in pretraining and downstream
  evaluation.
paper_authors._id: 656d3f168fb1ddf0d5549a5c
paper_authors.name: Albert Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:01:36.482Z
paper_authors._id: 642619b62864ea9e59de834e
paper_authors.avatarUrl: /avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
paper_authors.fullname: Albert Gu
paper_authors.type: user
paper_authors.user: albertgu
paper_authors._id: 656d3f168fb1ddf0d5549a5d
paper_authors.name: Tri Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:01:43.677Z
paper_authors._id: 64b8a6b5cf14c2fabe98159b
paper_authors.avatarUrl: /avatars/dbc009451865435bf290791beadc4723.svg
paper_authors.fullname: Tri Dao
paper_authors.type: user
paper_authors.user: tridao
paper_title: Mamba: Linear-Time Sequence Modeling with Selective State Spaces
paper_summary: Foundation models, now powering most of the exciting applications in deep
  learning, are almost universally based on the Transformer architecture and its
  core attention module. Many subquadratic-time architectures such as linear
  attention, gated convolution and recurrent models, and structured state space
  models (SSMs) have been developed to address Transformers' computational
  inefficiency on long sequences, but they have not performed as well as
  attention on important modalities such as language. We identify that a key
  weakness of such models is their inability to perform content-based reasoning,
  and make several improvements. First, simply letting the SSM parameters be
  functions of the input addresses their weakness with discrete modalities,
  allowing the model to selectively propagate or forget information along the
  sequence length dimension depending on the current token. Second, even though
  this change prevents the use of efficient convolutions, we design a
  hardware-aware parallel algorithm in recurrent mode. We integrate these
  selective SSMs into a simplified end-to-end neural network architecture without
  attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5times
  higher throughput than Transformers) and linear scaling in sequence length, and
  its performance improves on real data up to million-length sequences. As a
  general sequence model backbone, Mamba achieves state-of-the-art performance
  across several modalities such as language, audio, and genomics. On language
  modeling, our Mamba-3B model outperforms Transformers of the same size and
  matches Transformers twice its size, both in pretraining and downstream
  evaluation.
paper_ai_summary: Mamba, a novel SSM-based model, outperforms Transformers in inference speed and scalability across various modalities by selectively propagating information and using efficient hardware-aware algorithms.
paper_ai_keywords: Transformer architecture attention module linear attention gated convolution recurrent models structured state space models SSM content-based reasoning Mamba fast inference linear scaling sequence length genomics language modeling pretraining downstream evaluation