title: Selective Attention Improves Transformer
summary: Unneeded elements in the attention's context degrade performance. We
  introduce Selective Attention, a simple parameter-free change to the standard
  attention mechanism which reduces attention to unneeded elements. Selective
  attention improves language modeling performance in a variety of model sizes
  and context lengths. For example, a range of transformers trained with the
  language modeling objective on C4 with selective attention perform equivalently
  to standard transformers with ~2X more heads and parameters in their attention
  modules. Selective attention also allows decreasing the size of the attention's
  context buffer, leading to meaningful reductions in the memory and compute
  requirements during inference. For example, transformers with 100M parameters
  trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and
  47X less memory for their attention module, respectively, when equipped with
  selective attention, as those without selective attention, with the same
  validation perplexity.
paper_authors._id: 6700c824c67c24aeada0b186
paper_authors.name: Yaniv Leviathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T08:39:16.581Z
paper_authors._id: 6394a2244db1a9bedd1f2229
paper_authors.avatarUrl: /avatars/f04496811eb6873781e2ba606a2f97f0.svg
paper_authors.fullname: Yaniv Leviathan
paper_authors.type: user
paper_authors.user: yanivle
paper_authors._id: 6700c824c67c24aeada0b187
paper_authors.name: Matan Kalman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T15:10:22.649Z
paper_authors._id: 630414f64ec2dfa82a588fb8
paper_authors.avatarUrl: /avatars/a2c952fceacb2eab4484cd7f8015e931.svg
paper_authors.fullname: Matan Kalman
paper_authors.type: user
paper_authors.user: matank
paper_authors._id: 6700c824c67c24aeada0b188
paper_authors.name: Yossi Matias
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:36:23.526Z
paper_authors._id: 67281c9727f7e24422d156b1
paper_authors.avatarUrl: /avatars/207498b4733d5d7c8db8939248024d88.svg
paper_authors.fullname: Yossi Matias
paper_authors.type: user
paper_authors.user: yossimat
paper_title: Selective Attention Improves Transformer
paper_summary: Unneeded elements in the attention's context degrade performance. We
  introduce Selective Attention, a simple parameter-free change to the standard
  attention mechanism which reduces attention to unneeded elements. Selective
  attention improves language modeling performance in a variety of model sizes
  and context lengths. For example, a range of transformers trained with the
  language modeling objective on C4 with selective attention perform equivalently
  to standard transformers with ~2X more heads and parameters in their attention
  modules. Selective attention also allows decreasing the size of the attention's
  context buffer, leading to meaningful reductions in the memory and compute
  requirements during inference. For example, transformers with 100M parameters
  trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and
  47X less memory for their attention module, respectively, when equipped with
  selective attention, as those without selective attention, with the same
  validation perplexity.
paper_ai_keywords: Selective Attention attention mechanism language modeling transformers C4 attention heads parameters attention's context buffer memory compute requirements validation perplexity