title: SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference
summary: An efficient attention implementation is essential for large models due to
  its quadratic time complexity. Fortunately, attention commonly exhibits
  sparsity, i.e., many values in the attention map are near zero, allowing for
  the omission of corresponding computations. Many studies have utilized the
  sparse pattern to accelerate attention. However, most existing works focus on
  optimizing attention within specific models by exploiting certain sparse
  patterns of the attention map. A universal sparse attention that guarantees
  both the speedup and end-to-end performance of diverse models remains elusive.
  In this paper, we propose SpargeAttn, a universal sparse and quantized
  attention for any model. Our method uses a two-stage online filter: in the
  first stage, we rapidly and accurately predict the attention map, enabling the
  skip of some matrix multiplications in attention. In the second stage, we
  design an online softmax-aware filter that incurs no extra overhead and further
  skips some matrix multiplications. Experiments show that our method
  significantly accelerates diverse models, including language, image, and video
  generation, without sacrificing end-to-end metrics. The codes are available at
  https://github.com/thu-ml/SpargeAttn.
paper_authors._id: 67be8443ed8e258c0f70063a
paper_authors.name: Jintao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:25:57.704Z
paper_authors._id: 66c0a08bac74db25de8427ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
paper_authors.fullname: Jintao Zhang
paper_authors.type: user
paper_authors.user: jt-zhang
paper_authors._id: 67be8443ed8e258c0f70063b
paper_authors.name: Chendong Xiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T08:49:29.341Z
paper_authors._id: 6329bdbbde087eac2921e6a9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663679904323-noauth.jpeg
paper_authors.fullname: Xiangchendong
paper_authors.type: user
paper_authors.user: Xiang-cd
paper_authors._id: 67be8443ed8e258c0f70063c
paper_authors.name: Haofeng Huang
paper_authors._id: 67be8443ed8e258c0f70063d
paper_authors.name: Jia Wei
paper_authors._id: 67be8443ed8e258c0f70063e
paper_authors.name: Haocheng Xi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T08:49:45.446Z
paper_authors._id: 65d5a000ec7e31555e4db57e
paper_authors.avatarUrl: /avatars/aab8319fbaffdd53faff59a40ca5a5ea.svg
paper_authors.fullname: Haocheng Xi
paper_authors.type: user
paper_authors.user: hxi0408
paper_authors._id: 67be8443ed8e258c0f70063f
paper_authors.name: Jun Zhu
paper_authors._id: 67be8443ed8e258c0f700640
paper_authors.name: Jianfei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T08:49:52.550Z
paper_authors._id: 65fcad0ba0d7adc40b54fac2
paper_authors.avatarUrl: /avatars/7564b5642378fddb46ec3b5ae57c0402.svg
paper_authors.fullname: Jianfei Chen
paper_authors.type: user
paper_authors.user: surfingtomchen
paper_title: SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference
paper_summary: An efficient attention implementation is essential for large models due to
  its quadratic time complexity. Fortunately, attention commonly exhibits
  sparsity, i.e., many values in the attention map are near zero, allowing for
  the omission of corresponding computations. Many studies have utilized the
  sparse pattern to accelerate attention. However, most existing works focus on
  optimizing attention within specific models by exploiting certain sparse
  patterns of the attention map. A universal sparse attention that guarantees
  both the speedup and end-to-end performance of diverse models remains elusive.
  In this paper, we propose SpargeAttn, a universal sparse and quantized
  attention for any model. Our method uses a two-stage online filter: in the
  first stage, we rapidly and accurately predict the attention map, enabling the
  skip of some matrix multiplications in attention. In the second stage, we
  design an online softmax-aware filter that incurs no extra overhead and further
  skips some matrix multiplications. Experiments show that our method
  significantly accelerates diverse models, including language, image, and video
  generation, without sacrificing end-to-end metrics. The codes are available at
  https://github.com/thu-ml/SpargeAttn.
paper_ai_keywords: attention map matrix multiplications softmax-aware filter end-to-end performance SpargeAttn