title: Rectified Sparse Attention
summary: Efficient long-sequence generation is a critical challenge for Large Language
  Models. While recent sparse decoding methods improve efficiency, they suffer
  from KV cache misalignment, where approximation errors accumulate and degrade
  generation quality. In this work, we propose Rectified Sparse Attention (ReSA),
  a simple yet effective method that combines block-sparse attention with
  periodic dense rectification. By refreshing the KV cache at fixed intervals
  using a dense forward pass, ReSA bounds error accumulation and preserves
  alignment with the pretraining distribution. Experiments across math reasoning,
  language modeling, and retrieval tasks demonstrate that ReSA achieves
  near-lossless generation quality with significantly improved efficiency.
  Notably, ReSA delivers up to 2.42times end-to-end speedup under decoding at
  256K sequence length, making it a practical solution for scalable long-context
  inference. Code is available at https://aka.ms/ReSA-LM.
paper_authors._id: 684104a16b106ae42f5acc1a
paper_authors.name: Yutao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:54:25.439Z
paper_authors._id: 6300ef4779c5ddbc6cf83e1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661005591657-noauth.jpeg
paper_authors.fullname: Yutao Sun
paper_authors.type: user
paper_authors.user: sunyt32
paper_authors._id: 684104a16b106ae42f5acc1b
paper_authors.name: Tianzhu Ye
paper_authors._id: 684104a16b106ae42f5acc1c
paper_authors.name: Li Dong
paper_authors._id: 684104a16b106ae42f5acc1d
paper_authors.name: Yuqing Xia
paper_authors._id: 684104a16b106ae42f5acc1e
paper_authors.name: Jian Chen
paper_authors._id: 684104a16b106ae42f5acc1f
paper_authors.name: Yizhao Gao
paper_authors._id: 684104a16b106ae42f5acc20
paper_authors.name: Shijie Cao
paper_authors._id: 684104a16b106ae42f5acc21
paper_authors.name: Jianyong Wang
paper_authors._id: 684104a16b106ae42f5acc22
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:54:47.127Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Rectified Sparse Attention
paper_summary: Efficient long-sequence generation is a critical challenge for Large Language
  Models. While recent sparse decoding methods improve efficiency, they suffer
  from KV cache misalignment, where approximation errors accumulate and degrade
  generation quality. In this work, we propose Rectified Sparse Attention (ReSA),
  a simple yet effective method that combines block-sparse attention with
  periodic dense rectification. By refreshing the KV cache at fixed intervals
  using a dense forward pass, ReSA bounds error accumulation and preserves
  alignment with the pretraining distribution. Experiments across math reasoning,
  language modeling, and retrieval tasks demonstrate that ReSA achieves
  near-lossless generation quality with significantly improved efficiency.
  Notably, ReSA delivers up to 2.42times end-to-end speedup under decoding at
  256K sequence length, making it a practical solution for scalable long-context
  inference. Code is available at https://aka.ms/ReSA-LM.
paper_ai_summary: Rectified Sparse Attention (ReSA) improves the efficiency of long-sequence generation in Large Language Models by combining block-sparse attention with periodic dense rectification, maintaining high-quality generation.
paper_ai_keywords: sparse decoding KV cache misalignment Rectified Sparse Attention ReSA block-sparse attention dense rectification pretraining distribution long-context inference