title: Transformers are SSMs: Generalized Models and Efficient Algorithms
    Through Structured State Space Duality
summary: While Transformers have been the main architecture behind deep learning's
  success in language modeling, state-space models (SSMs) such as Mamba have
  recently been shown to match or outperform Transformers at small to medium
  scale. We show that these families of models are actually quite closely
  related, and develop a rich framework of theoretical connections between SSMs
  and variants of attention, connected through various decompositions of a
  well-studied class of structured semiseparable matrices. Our state space
  duality (SSD) framework allows us to design a new architecture (Mamba-2) whose
  core layer is an a refinement of Mamba's selective SSM that is 2-8X faster,
  while continuing to be competitive with Transformers on language modeling.
paper_authors._id: 665d3c002b4ea4231ce55b7c
paper_authors.name: Tri Dao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-17T17:59:39.666Z
paper_authors._id: 656ce060d848a6683aef9d34
paper_authors.avatarUrl: /avatars/0e5ec55ca725a5d4fff2937bfaa6640e.svg
paper_authors.fullname: Mamba
paper_authors.type: user
paper_authors.user: mamba-checkpoints
paper_authors._id: 665d3c002b4ea4231ce55b7d
paper_authors.name: Albert Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T07:13:19.811Z
paper_authors._id: 642619b62864ea9e59de834e
paper_authors.avatarUrl: /avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
paper_authors.fullname: Albert Gu
paper_authors.type: user
paper_authors.user: albertgu
paper_title: Transformers are SSMs: Generalized Models and Efficient Algorithms
    Through Structured State Space Duality
paper_summary: While Transformers have been the main architecture behind deep learning's
  success in language modeling, state-space models (SSMs) such as Mamba have
  recently been shown to match or outperform Transformers at small to medium
  scale. We show that these families of models are actually quite closely
  related, and develop a rich framework of theoretical connections between SSMs
  and variants of attention, connected through various decompositions of a
  well-studied class of structured semiseparable matrices. Our state space
  duality (SSD) framework allows us to design a new architecture (Mamba-2) whose
  core layer is an a refinement of Mamba's selective SSM that is 2-8X faster,
  while continuing to be competitive with Transformers on language modeling.
paper_ai_keywords: Transformers state-space models (SSMs) Mamba theoretical connections structured semiseparable matrices state space duality (SSD) Mamba-2 core layer selective SSM