SageAttention3: Microscaling FP4 Attention for Inference and An
  Exploration of 8-Bit Training
The efficiency of attention is important due to its quadratic time
complexity. We enhance the efficiency of attention through two key
contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to
accelerate attention computation. Our implementation achieves 1038 TOPS on
RTX5090, which is a 5x speedup over the fastest FlashAttention on RTX5090.
Experiments show that our FP4 attention can accelerate inference of various
models in a plug-and-play way. Second, we pioneer low-bit attention to training
tasks. Existing low-bit attention works like FlashAttention3 and SageAttention
focus only on inference. However, the efficiency of training large models is
also important. To explore whether low-bit attention can be effectively applied
to training tasks, we design an accurate and efficient 8-bit attention for both
forward and backward propagation. Experiments indicate that 8-bit attention
achieves lossless performance in fine-tuning tasks but exhibits slower
convergence in pretraining tasks. The code will be available at
https://github.com/thu-ml/SageAttention.
682d426251ce04237318cfe5
Jintao Zhang
claimed_verified
2025-05-21T08:40:46.065Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
user
jt-zhang
682d426251ce04237318cfe6
Jia Wei
682d426251ce04237318cfe7
Pengle Zhang
admin_assigned
2025-05-21T10:07:09.573Z
62cc11a4f1d37c16280a2923
/avatars/265b3cfb80f0a7b11a2ef67c49e29cf7.svg
Pengle Zhang
user
Guyan
682d426251ce04237318cfe8
Xiaoming Xu
682d426251ce04237318cfe9
Haofeng Huang
admin_assigned
2025-05-21T10:06:48.450Z
67ea1f6693f71dd8167a2d22
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/H_upra_XVG1AoBKUe9ArV.png
haofeng huang
user
haofeng666
682d426251ce04237318cfea
Haoxu Wang
claimed_verified
2025-05-21T10:30:12.064Z
658c1802a1105f8157ad1db9
https://cdn-avatars.huggingface.co/v1/production/uploads/658c1802a1105f8157ad1db9/WzjY29SkngxkKfiTYcssh.jpeg
whx1003
user
whx1003
682d426251ce04237318cfeb
Kai Jiang
682d426251ce04237318cfec
Jun Zhu
682d426251ce04237318cfed
Jianfei Chen
admin_assigned
2025-05-21T10:06:40.981Z
65fcad0ba0d7adc40b54fac2
/avatars/7564b5642378fddb46ec3b5ae57c0402.svg
Jianfei Chen
user
surfingtomchen
SageAttention3: Microscaling FP4 Attention for Inference and An
  Exploration of 8-Bit Training
The efficiency of attention is important due to its quadratic time
complexity. We enhance the efficiency of attention through two key
contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to
accelerate attention computation. Our implementation achieves 1038 TOPS on
RTX5090, which is a 5x speedup over the fastest FlashAttention on RTX5090.
Experiments show that our FP4 attention can accelerate inference of various
models in a plug-and-play way. Second, we pioneer low-bit attention to training
tasks. Existing low-bit attention works like FlashAttention3 and SageAttention
focus only on inference. However, the efficiency of training large models is
also important. To explore whether low-bit attention can be effectively applied
to training tasks, we design an accurate and efficient 8-bit attention for both
forward and backward propagation. Experiments indicate that 8-bit attention
achieves lossless performance in fine-tuning tasks but exhibits slower
convergence in pretraining tasks. The code will be available at
https://github.com/thu-ml/SageAttention.
Efficiency enhancements for attention mechanisms, including leveraging FP4 Tensor Cores and developing an 8-bit attention method, improve inference and training performance.
attention FP4 Tensor Cores Blackwell GPUs RTX5090 TOPS FlashAttention low-bit attention forward propagation backward propagation fine-tuning pretraining