SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference
  Acceleration
The transformer architecture predominates across various models. As the heart
of the transformer, attention has a computational complexity of O(N^2),
compared to O(N) for linear transformations. When handling large sequence
lengths, attention becomes the primary time-consuming component. Although
quantization has proven to be an effective method for accelerating model
inference, existing quantization methods primarily focus on optimizing the
linear layer. In response, we first analyze the feasibility of quantization in
attention detailedly. Following that, we propose SageAttention, a highly
efficient and accurate quantization method for attention. The OPS (operations
per second) of our approach outperforms FlashAttention2 and xformers by about
2.1 times and 2.7 times, respectively. SageAttention also achieves superior
accuracy performance over FlashAttention3. Comprehensive experiments confirm
that our approach incurs almost no end-to-end metrics loss across diverse
models, including those for large language processing, image generation, and
video generation.
66ff54bfda291ca087699440
Jintao Zhang
claimed_verified
2024-10-04T20:11:31.550Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
user
jt-zhang
66ff54bfda291ca087699441
Jia wei
66ff54bfda291ca087699442
Pengle Zhang
66ff54bfda291ca087699443
Jun Zhu
66ff54bfda291ca087699444
Jianfei Chen
admin_assigned
2024-10-04T09:08:18.301Z
65fcad0ba0d7adc40b54fac2
/avatars/7564b5642378fddb46ec3b5ae57c0402.svg
Jianfei Chen
user
surfingtomchen
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference
  Acceleration
The transformer architecture predominates across various models. As the heart
of the transformer, attention has a computational complexity of O(N^2),
compared to O(N) for linear transformations. When handling large sequence
lengths, attention becomes the primary time-consuming component. Although
quantization has proven to be an effective method for accelerating model
inference, existing quantization methods primarily focus on optimizing the
linear layer. In response, we first analyze the feasibility of quantization in
attention detailedly. Following that, we propose SageAttention, a highly
efficient and accurate quantization method for attention. The OPS (operations
per second) of our approach outperforms FlashAttention2 and xformers by about
2.1 times and 2.7 times, respectively. SageAttention also achieves superior
accuracy performance over FlashAttention3. Comprehensive experiments confirm
that our approach incurs almost no end-to-end metrics loss across diverse
models, including those for large language processing, image generation, and
video generation.
A new quantization method for attention mechanisms, named SageAttention, enhances inference speed and accuracy without significant loss in performance across various models.
transformer architecture attention computational complexity quantization SageAttention FlashAttention2 xformers large language processing image generation video generation