Gated Slot Attention for Efficient Linear-Time Sequence Modeling
Linear attention Transformers and their gated variants, celebrated for
enabling parallel training and efficient recurrent inference, still fall short
in recall-intensive tasks compared to traditional Transformers and demand
significant resources for training from scratch. This paper introduces Gated
Slot Attention (GSA), which enhances Attention with Bounded-memory-Control
(ABC) by incorporating a gating mechanism inspired by Gated Linear Attention
(GLA). Essentially, GSA comprises a two-layer GLA linked via softmax, utilizing
context-aware memory reading and adaptive forgetting to improve memory capacity
while maintaining compact recurrent state size. This design greatly enhances
both training and inference efficiency through GLA's hardware-efficient
training algorithm and reduced state size. Additionally, retaining the softmax
operation is particularly beneficial in "finetuning pretrained Transformers to
RNNs" (T2R) settings, reducing the need for extensive training from scratch.
Extensive experiments confirm GSA's superior performance in scenarios requiring
in-context recall and in T2R settings.
66e24c4c0fce83e241e894d3
Yu Zhang
extracted_confirmed
2024-09-12T03:42:33.714Z
638f5839f6de4b9e7e1627fb
https://cdn-avatars.huggingface.co/v1/production/uploads/638f5839f6de4b9e7e1627fb/6QGkrqRag6-GnH9k60Oil.jpeg
Yu Zhang
user
yzhangcs
66e24c4c0fce83e241e894d4
Songlin Yang
admin_assigned
2024-09-12T07:57:56.684Z
65810ad10d738799ecbce2de
https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
Songlin Yang
user
sonta7
66e24c4c0fce83e241e894d5
Ruijie Zhu
claimed_verified
2024-10-14T19:02:52.763Z
63ff09f24852102d4871c19c
https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
Rui-Jie Zhu
user
ridger
66e24c4c0fce83e241e894d6
Yue Zhang
66e24c4c0fce83e241e894d7
Leyang Cui
admin_assigned
2024-09-12T08:02:37.598Z
646db809e4fca2e115db428c
/avatars/17d77f59885db13d4f9c10a9b1245131.svg
Leyang Cui
user
nealcly
66e24c4c0fce83e241e894d8
Yiqiao Wang
66e24c4c0fce83e241e894d9
Bolun Wang
66e24c4c0fce83e241e894da
Freda Shi
admin_assigned
2024-09-12T08:03:40.556Z
6644f928b037ed8abd8a106c
https://cdn-avatars.huggingface.co/v1/production/uploads/6644f928b037ed8abd8a106c/KYfDDmihx1YUQCzpzWSOn.jpeg
Freda Shi
user
ExplorerFreda
66e24c4c0fce83e241e894db
Bailin Wang
66e24c4c0fce83e241e894dc
Wei Bi
66e24c4c0fce83e241e894dd
Peng Zhou
66e24c4c0fce83e241e894de
Guohong Fu
Gated Slot Attention for Efficient Linear-Time Sequence Modeling
Linear attention Transformers and their gated variants, celebrated for
enabling parallel training and efficient recurrent inference, still fall short
in recall-intensive tasks compared to traditional Transformers and demand
significant resources for training from scratch. This paper introduces Gated
Slot Attention (GSA), which enhances Attention with Bounded-memory-Control
(ABC) by incorporating a gating mechanism inspired by Gated Linear Attention
(GLA). Essentially, GSA comprises a two-layer GLA linked via softmax, utilizing
context-aware memory reading and adaptive forgetting to improve memory capacity
while maintaining compact recurrent state size. This design greatly enhances
both training and inference efficiency through GLA's hardware-efficient
training algorithm and reduced state size. Additionally, retaining the softmax
operation is particularly beneficial in "finetuning pretrained Transformers to
RNNs" (T2R) settings, reducing the need for extensive training from scratch.
Extensive experiments confirm GSA's superior performance in scenarios requiring
in-context recall and in T2R settings.
Gated Slot Attention (GSA) improves memory capacity and efficiency in Transformers through a bounded-memory-control gating mechanism, enhancing recall and facilitating efficient training and inference.
Linear attention Transformers gated variants parallel training recurrent inference Gated Slot Attention Bounded-memory-Control Gated Linear Attention context-aware memory softmax adaptive forgetting hardware-efficient training T2R settings