Taipan: Efficient and Expressive State Space Language Models with
  Selective Attention
Efficient long-context language modeling remains a significant challenge in
Natural Language Processing (NLP). While Transformers dominate language tasks,
they struggle with long sequences due to quadratic computational complexity in
training and linearly scaling memory costs during inference. Recent State Space
Models (SSMs) such as Mamba offer alternatives with constant memory usage, but
they underperform in tasks requiring extensive in-context retrieval. We
introduce Taipan, a novel hybrid architecture that combines Mamba-2 with
Selective Attention Layers (SALs). These SALs identify tokens requiring
long-range interactions, remove less important features, and then augment their
representations using the attention module. This approach balances Mamba's
efficiency with Transformer-like performance in memory-intensive tasks. By
constraining the attention budget, Taipan extends accurate predictions to
context lengths of up to 1 million tokens while preserving computational
efficiency. Our experiments demonstrate Taipan's superior performance across
various scales and tasks, offering a promising solution for efficient
long-context language modeling.
671b6fbb1bc6ee50c70a3e0c
Chien Van Nguyen
claimed_verified
2024-10-25T11:25:58.200Z
63edf17df5aef7e657262605
https://cdn-avatars.huggingface.co/v1/production/uploads/63edf17df5aef7e657262605/X3WzygIyND_0T_SWRDDU4.jpeg
Nguyen Van Chien
user
chiennv
671b6fbb1bc6ee50c70a3e0d
Huy Huu Nguyen
claimed_verified
2024-10-28T10:37:27.611Z
651a6b53757417990d5a3605
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QeVPeJ9av9MsEICRQl5Gk.jpeg
Nguyen Huu Huy
user
huy-nh-2000
671b6fbb1bc6ee50c70a3e0e
Thang M. Pham
671b6fbb1bc6ee50c70a3e0f
Ruiyi Zhang
claimed_verified
2024-11-05T20:17:35.335Z
64668f982da1abc242355cbb
/avatars/c9a248b8d70b1a8f7b78265c98690570.svg
Ruiyi Zhang
user
zhangry868
671b6fbb1bc6ee50c70a3e10
Hanieh Deilamsalehy
671b6fbb1bc6ee50c70a3e11
Puneet Mathur
671b6fbb1bc6ee50c70a3e12
Ryan A. Rossi
671b6fbb1bc6ee50c70a3e13
Trung Bui
claimed_verified
2024-10-30T09:54:45.099Z
6036eb46413a78f892e733a0
/avatars/b15925f89631323c176ecbd0eab99feb.svg
Trung Bui
user
Trung
671b6fbb1bc6ee50c70a3e14
Viet Dac Lai
claimed_verified
2024-10-28T10:37:24.318Z
63a0a29245edac9f75fc5d54
/avatars/67ba006edfd1ceddc54ea920c09c972b.svg
Viet Lai
user
laiviet
671b6fbb1bc6ee50c70a3e15
Franck Dernoncourt
claimed_verified
2024-10-28T10:37:26.031Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
671b6fbb1bc6ee50c70a3e16
Thien Huu Nguyen
Taipan: Efficient and Expressive State Space Language Models with
  Selective Attention
Efficient long-context language modeling remains a significant challenge in
Natural Language Processing (NLP). While Transformers dominate language tasks,
they struggle with long sequences due to quadratic computational complexity in
training and linearly scaling memory costs during inference. Recent State Space
Models (SSMs) such as Mamba offer alternatives with constant memory usage, but
they underperform in tasks requiring extensive in-context retrieval. We
introduce Taipan, a novel hybrid architecture that combines Mamba-2 with
Selective Attention Layers (SALs). These SALs identify tokens requiring
long-range interactions, remove less important features, and then augment their
representations using the attention module. This approach balances Mamba's
efficiency with Transformer-like performance in memory-intensive tasks. By
constraining the attention budget, Taipan extends accurate predictions to
context lengths of up to 1 million tokens while preserving computational
efficiency. Our experiments demonstrate Taipan's superior performance across
various scales and tasks, offering a promising solution for efficient
long-context language modeling.
Taipan combines Mamba-2 with Selective Attention Layers to achieve efficient long-context language modeling with high performance and constant memory usage.
Transformers State Space Models (SSMs) Mamba Selective Attention Layers (SALs) long-range interactions attention module long-context language modeling