SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
We introduce SeerAttention-R, a sparse attention framework specifically
tailored for the long decoding of reasoning models. Extended from
SeerAttention, SeerAttention-R retains the design of learning attention
sparsity through a self-distilled gating mechanism, while removing query
pooling to accommodate auto-regressive decoding. With a lightweight plug-in
gating, SeerAttention-R is flexible and can be easily integrated into existing
pretrained model without modifying the original parameters. We demonstrate that
SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning
accuracy with 4K token budget in AIME benchmark under large sparse attention
block sizes (64/128). Using TileLang, we develop a highly optimized sparse
decoding kernel that achieves near-theoretical speedups of up to 9x over
FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at:
https://github.com/microsoft/SeerAttention.
684a39599b38e1e5a33a6822
Yizhao Gao
684a39599b38e1e5a33a6823
Shuming Guo
684a39599b38e1e5a33a6824
Shijie Cao
684a39599b38e1e5a33a6825
Yuqing Xia
684a39599b38e1e5a33a6826
Yu Cheng
684a39599b38e1e5a33a6827
Lei Wang
684a39599b38e1e5a33a6828
Lingxiao Ma
684a39599b38e1e5a33a6829
Yutao Sun
684a39599b38e1e5a33a682a
Tianzhu Ye
684a39599b38e1e5a33a682b
Li Dong
684a39599b38e1e5a33a682c
Hayden Kwok-Hay So
684a39599b38e1e5a33a682d
Yu Hua
684a39599b38e1e5a33a682e
Ting Cao
684a39599b38e1e5a33a682f
Fan Yang
684a39599b38e1e5a33a6830
Mao Yang
SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
We introduce SeerAttention-R, a sparse attention framework specifically
tailored for the long decoding of reasoning models. Extended from
SeerAttention, SeerAttention-R retains the design of learning attention
sparsity through a self-distilled gating mechanism, while removing query
pooling to accommodate auto-regressive decoding. With a lightweight plug-in
gating, SeerAttention-R is flexible and can be easily integrated into existing
pretrained model without modifying the original parameters. We demonstrate that
SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning
accuracy with 4K token budget in AIME benchmark under large sparse attention
block sizes (64/128). Using TileLang, we develop a highly optimized sparse
decoding kernel that achieves near-theoretical speedups of up to 9x over
FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at:
https://github.com/microsoft/SeerAttention.
SeerAttention-R is a sparse attention framework for reasoning models that maintains high accuracy and achieves significant speedups through optimized sparse decoding kernels.
sparse attention reasoning models self-distilled gating mechanism query pooling lightweight plug-in gating AIME benchmark TileLang sparse decoding kernel FlashAttention-3 H100 GPU