title: SageAttention3: Microscaling FP4 Attention for Inference and An
    Exploration of 8-Bit Training
summary: The efficiency of attention is important due to its quadratic time
  complexity. We enhance the efficiency of attention through two key
  contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to
  accelerate attention computation. Our implementation achieves 1038 TOPS on
  RTX5090, which is a 5x speedup over the fastest FlashAttention on RTX5090.
  Experiments show that our FP4 attention can accelerate inference of various
  models in a plug-and-play way. Second, we pioneer low-bit attention to training
  tasks. Existing low-bit attention works like FlashAttention3 and SageAttention
  focus only on inference. However, the efficiency of training large models is
  also important. To explore whether low-bit attention can be effectively applied
  to training tasks, we design an accurate and efficient 8-bit attention for both
  forward and backward propagation. Experiments indicate that 8-bit attention
  achieves lossless performance in fine-tuning tasks but exhibits slower
  convergence in pretraining tasks. The code will be available at
  https://github.com/thu-ml/SageAttention.
paper_authors._id: 682d426251ce04237318cfe5
paper_authors.name: Jintao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:46.065Z
paper_authors._id: 66c0a08bac74db25de8427ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
paper_authors.fullname: Jintao Zhang
paper_authors.type: user
paper_authors.user: jt-zhang
paper_authors._id: 682d426251ce04237318cfe6
paper_authors.name: Jia Wei
paper_authors._id: 682d426251ce04237318cfe7
paper_authors.name: Pengle Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:07:09.573Z
paper_authors._id: 62cc11a4f1d37c16280a2923
paper_authors.avatarUrl: /avatars/265b3cfb80f0a7b11a2ef67c49e29cf7.svg
paper_authors.fullname: Pengle Zhang
paper_authors.type: user
paper_authors.user: Guyan
paper_authors._id: 682d426251ce04237318cfe8
paper_authors.name: Xiaoming Xu
paper_authors._id: 682d426251ce04237318cfe9
paper_authors.name: Haofeng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:06:48.450Z
paper_authors._id: 67ea1f6693f71dd8167a2d22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/H_upra_XVG1AoBKUe9ArV.png
paper_authors.fullname: haofeng huang
paper_authors.type: user
paper_authors.user: haofeng666
paper_authors._id: 682d426251ce04237318cfea
paper_authors.name: Haoxu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T10:30:12.064Z
paper_authors._id: 658c1802a1105f8157ad1db9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658c1802a1105f8157ad1db9/WzjY29SkngxkKfiTYcssh.jpeg
paper_authors.fullname: whx1003
paper_authors.type: user
paper_authors.user: whx1003
paper_authors._id: 682d426251ce04237318cfeb
paper_authors.name: Kai Jiang
paper_authors._id: 682d426251ce04237318cfec
paper_authors.name: Jun Zhu
paper_authors._id: 682d426251ce04237318cfed
paper_authors.name: Jianfei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:06:40.981Z
paper_authors._id: 65fcad0ba0d7adc40b54fac2
paper_authors.avatarUrl: /avatars/7564b5642378fddb46ec3b5ae57c0402.svg
paper_authors.fullname: Jianfei Chen
paper_authors.type: user
paper_authors.user: surfingtomchen
paper_title: SageAttention3: Microscaling FP4 Attention for Inference and An
    Exploration of 8-Bit Training
paper_summary: The efficiency of attention is important due to its quadratic time
  complexity. We enhance the efficiency of attention through two key
  contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to
  accelerate attention computation. Our implementation achieves 1038 TOPS on
  RTX5090, which is a 5x speedup over the fastest FlashAttention on RTX5090.
  Experiments show that our FP4 attention can accelerate inference of various
  models in a plug-and-play way. Second, we pioneer low-bit attention to training
  tasks. Existing low-bit attention works like FlashAttention3 and SageAttention
  focus only on inference. However, the efficiency of training large models is
  also important. To explore whether low-bit attention can be effectively applied
  to training tasks, we design an accurate and efficient 8-bit attention for both
  forward and backward propagation. Experiments indicate that 8-bit attention
  achieves lossless performance in fine-tuning tasks but exhibits slower
  convergence in pretraining tasks. The code will be available at
  https://github.com/thu-ml/SageAttention.
paper_ai_summary: Efficiency enhancements for attention mechanisms, including leveraging FP4 Tensor Cores and developing an 8-bit attention method, improve inference and training performance.
paper_ai_keywords: attention FP4 Tensor Cores Blackwell GPUs RTX5090 TOPS FlashAttention low-bit attention forward propagation backward propagation fine-tuning pretraining