title: SageAttention2 Technical Report: Accurate 4 Bit Attention for
    Plug-and-play Inference Acceleration
summary: Although quantization for linear layers has been widely used, its application
  to accelerate the attention process remains limited. SageAttention utilizes
  8-bit matrix multiplication, 16-bit matrix multiplication with 16-bit
  accumulator, and precision-enhancing methods, implementing an accurate and 2x
  speedup kernel compared to FlashAttention2. To further enhance the efficiency
  of attention computation while maintaining precision, we propose
  SageAttention2, which utilizes significantly faster 4-bit matrix multiplication
  (Matmul) alongside additional precision-enhancing techniques. First, we propose
  to quantize matrixes (Q, K) to INT4 in a warp-level granularity and quantize
  matrixes (widetilde P, V) to FP8. Second, we propose a method to smooth Q
  and V, enhancing the accuracy of attention with INT4 QK and FP8 PV.
  Third, we analyze the quantization accuracy across timesteps and layers, then
  propose an adaptive quantization method to ensure the end-to-end metrics over
  various models. The operations per second (OPS) of SageAttention2 surpass
  FlashAttention2 and xformers by about 3x and 5x on RTX4090, respectively.
  Comprehensive experiments confirm that our approach incurs negligible
  end-to-end metrics loss across diverse models, including those for large
  language processing, image generation, and video generation. The codes are
  available at https://github.com/thu-ml/SageAttention.
paper_authors._id: 673ea5022671f1feac00b6fd
paper_authors.name: Jintao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:37.368Z
paper_authors._id: 66c0a08bac74db25de8427ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
paper_authors.fullname: Jintao Zhang
paper_authors.type: user
paper_authors.user: jt-zhang
paper_authors._id: 673ea5022671f1feac00b6fe
paper_authors.name: Haofeng Huang
paper_authors._id: 673ea5022671f1feac00b6ff
paper_authors.name: Pengle Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:23:49.288Z
paper_authors._id: 62cc11a4f1d37c16280a2923
paper_authors.avatarUrl: /avatars/265b3cfb80f0a7b11a2ef67c49e29cf7.svg
paper_authors.fullname: Pengle Zhang
paper_authors.type: user
paper_authors.user: Guyan
paper_authors._id: 673ea5022671f1feac00b700
paper_authors.name: Jia Wei
paper_authors._id: 673ea5022671f1feac00b701
paper_authors.name: Jun Zhu
paper_authors._id: 673ea5022671f1feac00b702
paper_authors.name: Jianfei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:23:58.678Z
paper_authors._id: 65fcad0ba0d7adc40b54fac2
paper_authors.avatarUrl: /avatars/7564b5642378fddb46ec3b5ae57c0402.svg
paper_authors.fullname: Jianfei Chen
paper_authors.type: user
paper_authors.user: surfingtomchen
paper_title: SageAttention2 Technical Report: Accurate 4 Bit Attention for
    Plug-and-play Inference Acceleration
paper_summary: Although quantization for linear layers has been widely used, its application
  to accelerate the attention process remains limited. SageAttention utilizes
  8-bit matrix multiplication, 16-bit matrix multiplication with 16-bit
  accumulator, and precision-enhancing methods, implementing an accurate and 2x
  speedup kernel compared to FlashAttention2. To further enhance the efficiency
  of attention computation while maintaining precision, we propose
  SageAttention2, which utilizes significantly faster 4-bit matrix multiplication
  (Matmul) alongside additional precision-enhancing techniques. First, we propose
  to quantize matrixes (Q, K) to INT4 in a warp-level granularity and quantize
  matrixes (widetilde P, V) to FP8. Second, we propose a method to smooth Q
  and V, enhancing the accuracy of attention with INT4 QK and FP8 PV.
  Third, we analyze the quantization accuracy across timesteps and layers, then
  propose an adaptive quantization method to ensure the end-to-end metrics over
  various models. The operations per second (OPS) of SageAttention2 surpass
  FlashAttention2 and xformers by about 3x and 5x on RTX4090, respectively.
  Comprehensive experiments confirm that our approach incurs negligible
  end-to-end metrics loss across diverse models, including those for large
  language processing, image generation, and video generation. The codes are
  available at https://github.com/thu-ml/SageAttention.
paper_ai_summary: SageAttention2 improves attention computation speed with 4-bit matrix multiplication and adaptive quantization, achieving significant OPS increases without compromising accuracy.
paper_ai_keywords: SageAttention FlashAttention2 xformers matrix multiplication quantization INT4 FP8 smoothing adaptive quantization end-to-end metrics large language processing image generation video generation