XAttention: Block Sparse Attention with Antidiagonal Scoring
Long-Context Transformer Models (LCTMs) are vital for real-world applications
but suffer high computational costs due to attention's quadratic complexity.
Block-sparse attention mitigates this by focusing computation on critical
regions, yet existing methods struggle with balancing accuracy and efficiency
due to costly block importance measurements. In this paper, we introduce
XAttention, a plug-and-play framework that dramatically accelerates
long-context inference in Transformers models using sparse attention.
XAttention's key innovation is the insight that the sum of antidiagonal values
(i.e., from the lower-left to upper-right) in the attention matrix provides a
powerful proxy for block importance. This allows for precise identification and
pruning of non-essential blocks, resulting in high sparsity and dramatically
accelerated inference. Across comprehensive evaluations on demanding
long-context benchmarks-including RULER and LongBench for language, VideoMME
for video understanding, and VBench for video generation. XAttention achieves
accuracy comparable to full attention while delivering substantial
computational gains. We demonstrate up to 13.5x acceleration in attention
computation. These results underscore XAttention's ability to unlock the
practical potential of block sparse attention, paving the way for scalable and
efficient deployment of LCTMs in real-world applications. Code is available at
https://github.com/mit-han-lab/x-attention.
67dcd7a53c21e084fe58c3a8
Ruyi Xu
67dcd7a53c21e084fe58c3a9
Guangxuan Xiao
67dcd7a53c21e084fe58c3aa
Haofeng Huang
extracted_pending
2025-03-21T03:06:14.875Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
67dcd7a53c21e084fe58c3ab
Junxian Guo
67dcd7a53c21e084fe58c3ac
Song Han
XAttention: Block Sparse Attention with Antidiagonal Scoring
Long-Context Transformer Models (LCTMs) are vital for real-world applications
but suffer high computational costs due to attention's quadratic complexity.
Block-sparse attention mitigates this by focusing computation on critical
regions, yet existing methods struggle with balancing accuracy and efficiency
due to costly block importance measurements. In this paper, we introduce
XAttention, a plug-and-play framework that dramatically accelerates
long-context inference in Transformers models using sparse attention.
XAttention's key innovation is the insight that the sum of antidiagonal values
(i.e., from the lower-left to upper-right) in the attention matrix provides a
powerful proxy for block importance. This allows for precise identification and
pruning of non-essential blocks, resulting in high sparsity and dramatically
accelerated inference. Across comprehensive evaluations on demanding
long-context benchmarks-including RULER and LongBench for language, VideoMME
for video understanding, and VBench for video generation. XAttention achieves
accuracy comparable to full attention while delivering substantial
computational gains. We demonstrate up to 13.5x acceleration in attention
computation. These results underscore XAttention's ability to unlock the
practical potential of block sparse attention, paving the way for scalable and
efficient deployment of LCTMs in real-world applications. Code is available at
https://github.com/mit-han-lab/x-attention.
XAttention is a framework that uses antidiagonal sums in the attention matrix to efficiently identify and prune unimportant blocks in Transformers, achieving high accuracy with significant computational savings.
Long-Context Transformer Models (LCTMs) block-sparse attention antidiagonal values attention matrix XAttention sparse attention RULER LongBench VideoMME VBench video understanding video generation