title: Delta Attention: Fast and Accurate Sparse Attention Inference by Delta
    Correction
summary: The attention mechanism of a transformer has a quadratic complexity, leading
  to high inference costs and latency for long sequences. However, attention
  matrices are mostly sparse, which implies that many entries may be omitted from
  computation for efficient inference. Sparse attention inference methods aim to
  reduce this computational burden; however, they also come with a troublesome
  performance degradation. We discover that one reason for this degradation is
  that the sparse calculation induces a distributional shift in the attention
  outputs. The distributional shift causes decoding-time queries to fail to align
  well with the appropriate keys from the prefill stage, leading to a drop in
  performance. We propose a simple, novel, and effective procedure for correcting
  this distributional shift, bringing the distribution of sparse attention
  outputs closer to that of quadratic attention. Our method can be applied on top
  of any sparse attention method, and results in an average 36%pt performance
  increase, recovering 88% of quadratic attention accuracy on the 131K RULER
  benchmark when applied on top of sliding window attention with sink tokens
  while only adding a small overhead. Our method can maintain approximately 98.5%
  sparsity over full quadratic attention, making our model 32 times faster than
  Flash Attention 2 when processing 1M token prefills.
paper_authors._id: 682bf899ca2c97f999864e23
paper_authors.name: Jeffrey Willette
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:26.628Z
paper_authors._id: 654c5d6548b4741202739b73
paper_authors.avatarUrl: /avatars/bf1bfcf34d93136b7d3a48cebf014d45.svg
paper_authors.fullname: Jeff Willette
paper_authors.type: user
paper_authors.user: jeffwillette
paper_authors._id: 682bf899ca2c97f999864e24
paper_authors.name: Heejun Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:28.954Z
paper_authors._id: 62e622d08e0b2dc6707f8794
paper_authors.avatarUrl: /avatars/8c47b5c862f82d4258ba707c932f7f87.svg
paper_authors.fullname: Heejun Lee
paper_authors.type: user
paper_authors.user: gmlwns5176
paper_authors._id: 682bf899ca2c97f999864e25
paper_authors.name: Sung Ju Hwang
paper_title: Delta Attention: Fast and Accurate Sparse Attention Inference by Delta
    Correction
paper_summary: The attention mechanism of a transformer has a quadratic complexity, leading
  to high inference costs and latency for long sequences. However, attention
  matrices are mostly sparse, which implies that many entries may be omitted from
  computation for efficient inference. Sparse attention inference methods aim to
  reduce this computational burden; however, they also come with a troublesome
  performance degradation. We discover that one reason for this degradation is
  that the sparse calculation induces a distributional shift in the attention
  outputs. The distributional shift causes decoding-time queries to fail to align
  well with the appropriate keys from the prefill stage, leading to a drop in
  performance. We propose a simple, novel, and effective procedure for correcting
  this distributional shift, bringing the distribution of sparse attention
  outputs closer to that of quadratic attention. Our method can be applied on top
  of any sparse attention method, and results in an average 36%pt performance
  increase, recovering 88% of quadratic attention accuracy on the 131K RULER
  benchmark when applied on top of sliding window attention with sink tokens
  while only adding a small overhead. Our method can maintain approximately 98.5%
  sparsity over full quadratic attention, making our model 32 times faster than
  Flash Attention 2 when processing 1M token prefills.
paper_ai_summary: A novel procedure corrects distributional shifts in sparse attention, enhancing performance and reducing computational cost compared to full quadratic attention.
paper_ai_keywords: attention mechanism transformer quadratic complexity inference costs latency sparse attention distributional shift decoding-time queries keys prefill stage sliding window attention sink tokens Flash Attention 2