Selective Attention Improves Transformer
Unneeded elements in the attention's context degrade performance. We
introduce Selective Attention, a simple parameter-free change to the standard
attention mechanism which reduces attention to unneeded elements. Selective
attention improves language modeling performance in a variety of model sizes
and context lengths. For example, a range of transformers trained with the
language modeling objective on C4 with selective attention perform equivalently
to standard transformers with ~2X more heads and parameters in their attention
modules. Selective attention also allows decreasing the size of the attention's
context buffer, leading to meaningful reductions in the memory and compute
requirements during inference. For example, transformers with 100M parameters
trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and
47X less memory for their attention module, respectively, when equipped with
selective attention, as those without selective attention, with the same
validation perplexity.
6700c824c67c24aeada0b186
Yaniv Leviathan
admin_assigned
2024-10-07T08:39:16.581Z
6394a2244db1a9bedd1f2229
/avatars/f04496811eb6873781e2ba606a2f97f0.svg
Yaniv Leviathan
user
yanivle
6700c824c67c24aeada0b187
Matan Kalman
claimed_verified
2024-10-07T15:10:22.649Z
630414f64ec2dfa82a588fb8
/avatars/a2c952fceacb2eab4484cd7f8015e931.svg
Matan Kalman
user
matank
6700c824c67c24aeada0b188
Yossi Matias
claimed_verified
2024-11-04T09:36:23.526Z
67281c9727f7e24422d156b1
/avatars/207498b4733d5d7c8db8939248024d88.svg
Yossi Matias
user
yossimat
Selective Attention Improves Transformer
Unneeded elements in the attention's context degrade performance. We
introduce Selective Attention, a simple parameter-free change to the standard
attention mechanism which reduces attention to unneeded elements. Selective
attention improves language modeling performance in a variety of model sizes
and context lengths. For example, a range of transformers trained with the
language modeling objective on C4 with selective attention perform equivalently
to standard transformers with ~2X more heads and parameters in their attention
modules. Selective attention also allows decreasing the size of the attention's
context buffer, leading to meaningful reductions in the memory and compute
requirements during inference. For example, transformers with 100M parameters
trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and
47X less memory for their attention module, respectively, when equipped with
selective attention, as those without selective attention, with the same
validation perplexity.
Selective Attention reduces unnecessary context elements in the attention mechanism, improving language modeling performance and decreasing memory and compute requirements.
Selective Attention standard attention mechanism language modeling transformers C4 attention heads context buffer memory requirements compute requirements validation perplexity