title: XAttention: Block Sparse Attention with Antidiagonal Scoring
summary: Long-Context Transformer Models (LCTMs) are vital for real-world applications
  but suffer high computational costs due to attention's quadratic complexity.
  Block-sparse attention mitigates this by focusing computation on critical
  regions, yet existing methods struggle with balancing accuracy and efficiency
  due to costly block importance measurements. In this paper, we introduce
  XAttention, a plug-and-play framework that dramatically accelerates
  long-context inference in Transformers models using sparse attention.
  XAttention's key innovation is the insight that the sum of antidiagonal values
  (i.e., from the lower-left to upper-right) in the attention matrix provides a
  powerful proxy for block importance. This allows for precise identification and
  pruning of non-essential blocks, resulting in high sparsity and dramatically
  accelerated inference. Across comprehensive evaluations on demanding
  long-context benchmarks-including RULER and LongBench for language, VideoMME
  for video understanding, and VBench for video generation. XAttention achieves
  accuracy comparable to full attention while delivering substantial
  computational gains. We demonstrate up to 13.5x acceleration in attention
  computation. These results underscore XAttention's ability to unlock the
  practical potential of block sparse attention, paving the way for scalable and
  efficient deployment of LCTMs in real-world applications. Code is available at
  https://github.com/mit-han-lab/x-attention.
paper_authors._id: 67dcd7a53c21e084fe58c3a8
paper_authors.name: Ruyi Xu
paper_authors._id: 67dcd7a53c21e084fe58c3a9
paper_authors.name: Guangxuan Xiao
paper_authors._id: 67dcd7a53c21e084fe58c3aa
paper_authors.name: Haofeng Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-21T03:06:14.875Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_authors._id: 67dcd7a53c21e084fe58c3ab
paper_authors.name: Junxian Guo
paper_authors._id: 67dcd7a53c21e084fe58c3ac
paper_authors.name: Song Han
paper_title: XAttention: Block Sparse Attention with Antidiagonal Scoring
paper_summary: Long-Context Transformer Models (LCTMs) are vital for real-world applications
  but suffer high computational costs due to attention's quadratic complexity.
  Block-sparse attention mitigates this by focusing computation on critical
  regions, yet existing methods struggle with balancing accuracy and efficiency
  due to costly block importance measurements. In this paper, we introduce
  XAttention, a plug-and-play framework that dramatically accelerates
  long-context inference in Transformers models using sparse attention.
  XAttention's key innovation is the insight that the sum of antidiagonal values
  (i.e., from the lower-left to upper-right) in the attention matrix provides a
  powerful proxy for block importance. This allows for precise identification and
  pruning of non-essential blocks, resulting in high sparsity and dramatically
  accelerated inference. Across comprehensive evaluations on demanding
  long-context benchmarks-including RULER and LongBench for language, VideoMME
  for video understanding, and VBench for video generation. XAttention achieves
  accuracy comparable to full attention while delivering substantial
  computational gains. We demonstrate up to 13.5x acceleration in attention
  computation. These results underscore XAttention's ability to unlock the
  practical potential of block sparse attention, paving the way for scalable and
  efficient deployment of LCTMs in real-world applications. Code is available at
  https://github.com/mit-han-lab/x-attention.
paper_ai_summary: XAttention is a framework that uses antidiagonal sums in the attention matrix to efficiently identify and prune unimportant blocks in Transformers, achieving high accuracy with significant computational savings.
paper_ai_keywords: Long-Context Transformer Models (LCTMs) block-sparse attention antidiagonal values attention matrix XAttention sparse attention RULER LongBench VideoMME VBench video understanding video generation