title: Gated Slot Attention for Efficient Linear-Time Sequence Modeling
summary: Linear attention Transformers and their gated variants, celebrated for
  enabling parallel training and efficient recurrent inference, still fall short
  in recall-intensive tasks compared to traditional Transformers and demand
  significant resources for training from scratch. This paper introduces Gated
  Slot Attention (GSA), which enhances Attention with Bounded-memory-Control
  (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention
  (GLA). Essentially, GSA comprises a two-layer GLA linked via softmax, utilizing
  context-aware memory reading and adaptive forgetting to improve memory capacity
  while maintaining compact recurrent state size. This design greatly enhances
  both training and inference efficiency through GLA's hardware-efficient
  training algorithm and reduced state size. Additionally, retaining the softmax
  operation is particularly beneficial in "finetuning pretrained Transformers to
  RNNs" (T2R) settings, reducing the need for extensive training from scratch.
  Extensive experiments confirm GSA's superior performance in scenarios requiring
  in-context recall and in T2R settings.
paper_authors._id: 66e24c4c0fce83e241e894d3
paper_authors.name: Yu Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-12T03:42:33.714Z
paper_authors._id: 638f5839f6de4b9e7e1627fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638f5839f6de4b9e7e1627fb/6QGkrqRag6-GnH9k60Oil.jpeg
paper_authors.fullname: Yu Zhang
paper_authors.type: user
paper_authors.user: yzhangcs
paper_authors._id: 66e24c4c0fce83e241e894d4
paper_authors.name: Songlin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:57:56.684Z
paper_authors._id: 65810ad10d738799ecbce2de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
paper_authors.fullname: Songlin Yang
paper_authors.type: user
paper_authors.user: sonta7
paper_authors._id: 66e24c4c0fce83e241e894d5
paper_authors.name: Ruijie Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:02:52.763Z
paper_authors._id: 63ff09f24852102d4871c19c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
paper_authors.fullname: Rui-Jie Zhu
paper_authors.type: user
paper_authors.user: ridger
paper_authors._id: 66e24c4c0fce83e241e894d6
paper_authors.name: Yue Zhang
paper_authors._id: 66e24c4c0fce83e241e894d7
paper_authors.name: Leyang Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T08:02:37.598Z
paper_authors._id: 646db809e4fca2e115db428c
paper_authors.avatarUrl: /avatars/17d77f59885db13d4f9c10a9b1245131.svg
paper_authors.fullname: Leyang Cui
paper_authors.type: user
paper_authors.user: nealcly
paper_authors._id: 66e24c4c0fce83e241e894d8
paper_authors.name: Yiqiao Wang
paper_authors._id: 66e24c4c0fce83e241e894d9
paper_authors.name: Bolun Wang
paper_authors._id: 66e24c4c0fce83e241e894da
paper_authors.name: Freda Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T08:03:40.556Z
paper_authors._id: 6644f928b037ed8abd8a106c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6644f928b037ed8abd8a106c/KYfDDmihx1YUQCzpzWSOn.jpeg
paper_authors.fullname: Freda Shi
paper_authors.type: user
paper_authors.user: ExplorerFreda
paper_authors._id: 66e24c4c0fce83e241e894db
paper_authors.name: Bailin Wang
paper_authors._id: 66e24c4c0fce83e241e894dc
paper_authors.name: Wei Bi
paper_authors._id: 66e24c4c0fce83e241e894dd
paper_authors.name: Peng Zhou
paper_authors._id: 66e24c4c0fce83e241e894de
paper_authors.name: Guohong Fu
paper_title: Gated Slot Attention for Efficient Linear-Time Sequence Modeling
paper_summary: Linear attention Transformers and their gated variants, celebrated for
  enabling parallel training and efficient recurrent inference, still fall short
  in recall-intensive tasks compared to traditional Transformers and demand
  significant resources for training from scratch. This paper introduces Gated
  Slot Attention (GSA), which enhances Attention with Bounded-memory-Control
  (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention
  (GLA). Essentially, GSA comprises a two-layer GLA linked via softmax, utilizing
  context-aware memory reading and adaptive forgetting to improve memory capacity
  while maintaining compact recurrent state size. This design greatly enhances
  both training and inference efficiency through GLA's hardware-efficient
  training algorithm and reduced state size. Additionally, retaining the softmax
  operation is particularly beneficial in "finetuning pretrained Transformers to
  RNNs" (T2R) settings, reducing the need for extensive training from scratch.
  Extensive experiments confirm GSA's superior performance in scenarios requiring
  in-context recall and in T2R settings.
paper_ai_summary: Gated Slot Attention (GSA) improves memory capacity and efficiency in Transformers through a bounded-memory-control gating mechanism, enhancing recall and facilitating efficient training and inference.
paper_ai_keywords: Linear attention Transformers gated variants parallel training recurrent inference Gated Slot Attention Bounded-memory-Control Gated Linear Attention context-aware memory softmax adaptive forgetting hardware-efficient training T2R settings