title: Taipan: Efficient and Expressive State Space Language Models with
    Selective Attention
summary: Efficient long-context language modeling remains a significant challenge in
  Natural Language Processing (NLP). While Transformers dominate language tasks,
  they struggle with long sequences due to quadratic computational complexity in
  training and linearly scaling memory costs during inference. Recent State Space
  Models (SSMs) such as Mamba offer alternatives with constant memory usage, but
  they underperform in tasks requiring extensive in-context retrieval. We
  introduce Taipan, a novel hybrid architecture that combines Mamba-2 with
  Selective Attention Layers (SALs). These SALs identify tokens requiring
  long-range interactions, remove less important features, and then augment their
  representations using the attention module. This approach balances Mamba's
  efficiency with Transformer-like performance in memory-intensive tasks. By
  constraining the attention budget, Taipan extends accurate predictions to
  context lengths of up to 1 million tokens while preserving computational
  efficiency. Our experiments demonstrate Taipan's superior performance across
  various scales and tasks, offering a promising solution for efficient
  long-context language modeling.
paper_authors._id: 671b6fbb1bc6ee50c70a3e0c
paper_authors.name: Chien Van Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T11:25:58.200Z
paper_authors._id: 63edf17df5aef7e657262605
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63edf17df5aef7e657262605/X3WzygIyND_0T_SWRDDU4.jpeg
paper_authors.fullname: Nguyen Van Chien
paper_authors.type: user
paper_authors.user: chiennv
paper_authors._id: 671b6fbb1bc6ee50c70a3e0d
paper_authors.name: Huy Huu Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:37:27.611Z
paper_authors._id: 651a6b53757417990d5a3605
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QeVPeJ9av9MsEICRQl5Gk.jpeg
paper_authors.fullname: Nguyen Huu Huy
paper_authors.type: user
paper_authors.user: huy-nh-2000
paper_authors._id: 671b6fbb1bc6ee50c70a3e0e
paper_authors.name: Thang M. Pham
paper_authors._id: 671b6fbb1bc6ee50c70a3e0f
paper_authors.name: Ruiyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T20:17:35.335Z
paper_authors._id: 64668f982da1abc242355cbb
paper_authors.avatarUrl: /avatars/c9a248b8d70b1a8f7b78265c98690570.svg
paper_authors.fullname: Ruiyi Zhang
paper_authors.type: user
paper_authors.user: zhangry868
paper_authors._id: 671b6fbb1bc6ee50c70a3e10
paper_authors.name: Hanieh Deilamsalehy
paper_authors._id: 671b6fbb1bc6ee50c70a3e11
paper_authors.name: Puneet Mathur
paper_authors._id: 671b6fbb1bc6ee50c70a3e12
paper_authors.name: Ryan A. Rossi
paper_authors._id: 671b6fbb1bc6ee50c70a3e13
paper_authors.name: Trung Bui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:45.099Z
paper_authors._id: 6036eb46413a78f892e733a0
paper_authors.avatarUrl: /avatars/b15925f89631323c176ecbd0eab99feb.svg
paper_authors.fullname: Trung Bui
paper_authors.type: user
paper_authors.user: Trung
paper_authors._id: 671b6fbb1bc6ee50c70a3e14
paper_authors.name: Viet Dac Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:37:24.318Z
paper_authors._id: 63a0a29245edac9f75fc5d54
paper_authors.avatarUrl: /avatars/67ba006edfd1ceddc54ea920c09c972b.svg
paper_authors.fullname: Viet Lai
paper_authors.type: user
paper_authors.user: laiviet
paper_authors._id: 671b6fbb1bc6ee50c70a3e15
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:37:26.031Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 671b6fbb1bc6ee50c70a3e16
paper_authors.name: Thien Huu Nguyen
paper_title: Taipan: Efficient and Expressive State Space Language Models with
    Selective Attention
paper_summary: Efficient long-context language modeling remains a significant challenge in
  Natural Language Processing (NLP). While Transformers dominate language tasks,
  they struggle with long sequences due to quadratic computational complexity in
  training and linearly scaling memory costs during inference. Recent State Space
  Models (SSMs) such as Mamba offer alternatives with constant memory usage, but
  they underperform in tasks requiring extensive in-context retrieval. We
  introduce Taipan, a novel hybrid architecture that combines Mamba-2 with
  Selective Attention Layers (SALs). These SALs identify tokens requiring
  long-range interactions, remove less important features, and then augment their
  representations using the attention module. This approach balances Mamba's
  efficiency with Transformer-like performance in memory-intensive tasks. By
  constraining the attention budget, Taipan extends accurate predictions to
  context lengths of up to 1 million tokens while preserving computational
  efficiency. Our experiments demonstrate Taipan's superior performance across
  various scales and tasks, offering a promising solution for efficient
  long-context language modeling.
paper_ai_keywords: Transformers long sequences computational complexity memory costs State Space Models (SSMs) Mamba Taipan Selective Attention Layers (SALs) long-range networks attention module memory-intensive tasks context triples