title: SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
summary: Attention is the cornerstone of modern Large Language Models (LLMs). Yet its
  quadratic complexity limits the efficiency and scalability of LLMs, especially
  for those with a long-context window. A promising approach addressing this
  limitation is to leverage the sparsity in attention. However, existing
  sparsity-based solutions predominantly rely on predefined patterns or
  heuristics to approximate sparsity. This practice falls short to fully capture
  the dynamic nature of attention sparsity in language-based tasks. This paper
  argues that attention sparsity should be learned rather than predefined. To
  this end, we design SeerAttention, a new Attention mechanism that augments the
  conventional attention with a learnable gate that adaptively selects
  significant blocks in an attention map and deems the rest blocks sparse. Such
  block-level sparsity effectively balances accuracy and speedup. To enable
  efficient learning of the gating network, we develop a customized
  FlashAttention implementation that extracts the block-level ground truth of
  attention map with minimum overhead. SeerAttention not only applies to
  post-training, but also excels in long-context fine-tuning. Our results show
  that at post-training stages, SeerAttention significantly outperforms
  state-of-the-art static or heuristic-based sparse attention methods, while also
  being more versatile and flexible to adapt to varying context lengths and
  sparsity ratios. When applied to long-context fine-tuning with YaRN,
  SeerAttention can achieve a remarkable 90% sparsity ratio at a 32k context
  length with minimal perplexity loss, offering a 5.67x speedup over
  FlashAttention-2.
paper_authors._id: 6715f1f89c2303f000b006b6
paper_authors.name: Yizhao Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:40:49.287Z
paper_authors._id: 661c96f48921f03a9dae04c3
paper_authors.avatarUrl: /avatars/c486a45ffea7d1a8c72bd8512014b07e.svg
paper_authors.fullname: Yizhao Gao
paper_authors.type: user
paper_authors.user: LongMountain
paper_authors._id: 6715f1f89c2303f000b006b7
paper_authors.name: Zhichen Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:40:52.311Z
paper_authors._id: 66038f8e90c62bb38f06eb9f
paper_authors.avatarUrl: /avatars/e67137b47f45f580bae7ea6472671359.svg
paper_authors.fullname: Zhichen Zeng
paper_authors.type: user
paper_authors.user: CharyZeng
paper_authors._id: 6715f1f89c2303f000b006b8
paper_authors.name: Dayou Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:08:32.611Z
paper_authors._id: 64551fcba13edf669cd84a88
paper_authors.avatarUrl: /avatars/c3c9170d068757f610280518c2baf443.svg
paper_authors.fullname: Dayou DU
paper_authors.type: user
paper_authors.user: Daniel-Duda
paper_authors._id: 6715f1f89c2303f000b006b9
paper_authors.name: Shijie Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:08:50.992Z
paper_authors._id: 662852bf8bfc90408a97acdf
paper_authors.avatarUrl: /avatars/008e6faf51e1be4f0c0d17008dfe2f49.svg
paper_authors.fullname: cao
paper_authors.type: user
paper_authors.user: tingcao
paper_authors._id: 6715f1f89c2303f000b006ba
paper_authors.name: Hayden Kwok-Hay So
paper_authors._id: 6715f1f89c2303f000b006bb
paper_authors.name: Ting Cao
paper_authors._id: 6715f1f89c2303f000b006bc
paper_authors.name: Fan Yang
paper_authors._id: 6715f1f89c2303f000b006bd
paper_authors.name: Mao Yang
paper_title: SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
paper_summary: Attention is the cornerstone of modern Large Language Models (LLMs). Yet its
  quadratic complexity limits the efficiency and scalability of LLMs, especially
  for those with a long-context window. A promising approach addressing this
  limitation is to leverage the sparsity in attention. However, existing
  sparsity-based solutions predominantly rely on predefined patterns or
  heuristics to approximate sparsity. This practice falls short to fully capture
  the dynamic nature of attention sparsity in language-based tasks. This paper
  argues that attention sparsity should be learned rather than predefined. To
  this end, we design SeerAttention, a new Attention mechanism that augments the
  conventional attention with a learnable gate that adaptively selects
  significant blocks in an attention map and deems the rest blocks sparse. Such
  block-level sparsity effectively balances accuracy and speedup. To enable
  efficient learning of the gating network, we develop a customized
  FlashAttention implementation that extracts the block-level ground truth of
  attention map with minimum overhead. SeerAttention not only applies to
  post-training, but also excels in long-context fine-tuning. Our results show
  that at post-training stages, SeerAttention significantly outperforms
  state-of-the-art static or heuristic-based sparse attention methods, while also
  being more versatile and flexible to adapt to varying context lengths and
  sparsity ratios. When applied to long-context fine-tuning with YaRN,
  SeerAttention can achieve a remarkable 90% sparsity ratio at a 32k context
  length with minimal perplexity loss, offering a 5.67x speedup over
  FlashAttention-2.
paper_ai_summary: SeerAttention introduces a learnable attention mechanism that adaptsively selects significant blocks in attention maps, balancing accuracy and speed, and achieves superior performance over static or heuristic-based sparse attention methods.
paper_ai_keywords: attention Large Language Models LLMs quadratic complexity sparsity attention sparsity SeerAttention gating network block-level sparsity FlashAttention long-context fine-tuning YaRN perplexity loss speedup