Transformers are SSMs: Generalized Models and Efficient Algorithms
  Through Structured State Space Duality
While Transformers have been the main architecture behind deep learning's
success in language modeling, state-space models (SSMs) such as Mamba have
recently been shown to match or outperform Transformers at small to medium
scale. We show that these families of models are actually quite closely
related, and develop a rich framework of theoretical connections between SSMs
and variants of attention, connected through various decompositions of a
well-studied class of structured semiseparable matrices. Our state space
duality (SSD) framework allows us to design a new architecture (Mamba-2) whose
core layer is an a refinement of Mamba's selective SSM that is 2-8X faster,
while continuing to be competitive with Transformers on language modeling.
665d3c002b4ea4231ce55b7c
Tri Dao
extracted_confirmed
2024-07-17T17:59:39.666Z
656ce060d848a6683aef9d34
/avatars/0e5ec55ca725a5d4fff2937bfaa6640e.svg
Mamba
user
mamba-checkpoints
665d3c002b4ea4231ce55b7d
Albert Gu
admin_assigned
2024-06-03T07:13:19.811Z
642619b62864ea9e59de834e
/avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
Albert Gu
user
albertgu
Transformers are SSMs: Generalized Models and Efficient Algorithms
  Through Structured State Space Duality
While Transformers have been the main architecture behind deep learning's
success in language modeling, state-space models (SSMs) such as Mamba have
recently been shown to match or outperform Transformers at small to medium
scale. We show that these families of models are actually quite closely
related, and develop a rich framework of theoretical connections between SSMs
and variants of attention, connected through various decompositions of a
well-studied class of structured semiseparable matrices. Our state space
duality (SSD) framework allows us to design a new architecture (Mamba-2) whose
core layer is an a refinement of Mamba's selective SSM that is 2-8X faster,
while continuing to be competitive with Transformers on language modeling.
A new framework connects state-space models and transformers, leading to a faster architecture for language modeling.
Transformers state-space models Mamba selective SSM state space duality structured semiseparable matrices Mamba-2