title: SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference
    Acceleration
summary: The transformer architecture predominates across various models. As the heart
  of the transformer, attention has a computational complexity of O(N^2),
  compared to O(N) for linear transformations. When handling large sequence
  lengths, attention becomes the primary time-consuming component. Although
  quantization has proven to be an effective method for accelerating model
  inference, existing quantization methods primarily focus on optimizing the
  linear layer. In response, we first analyze the feasibility of quantization in
  attention detailedly. Following that, we propose SageAttention, a highly
  efficient and accurate quantization method for attention. The OPS (operations
  per second) of our approach outperforms FlashAttention2 and xformers by about
  2.1 times and 2.7 times, respectively. SageAttention also achieves superior
  accuracy performance over FlashAttention3. Comprehensive experiments confirm
  that our approach incurs almost no end-to-end metrics loss across diverse
  models, including those for large language processing, image generation, and
  video generation.
paper_authors._id: 66ff54bfda291ca087699440
paper_authors.name: Jintao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T20:11:31.550Z
paper_authors._id: 66c0a08bac74db25de8427ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
paper_authors.fullname: Jintao Zhang
paper_authors.type: user
paper_authors.user: jt-zhang
paper_authors._id: 66ff54bfda291ca087699441
paper_authors.name: Jia wei
paper_authors._id: 66ff54bfda291ca087699442
paper_authors.name: Pengle Zhang
paper_authors._id: 66ff54bfda291ca087699443
paper_authors.name: Jun Zhu
paper_authors._id: 66ff54bfda291ca087699444
paper_authors.name: Jianfei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:08:18.301Z
paper_authors._id: 65fcad0ba0d7adc40b54fac2
paper_authors.avatarUrl: /avatars/7564b5642378fddb46ec3b5ae57c0402.svg
paper_authors.fullname: Jianfei Chen
paper_authors.type: user
paper_authors.user: surfingtomchen
paper_title: SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference
    Acceleration
paper_summary: The transformer architecture predominates across various models. As the heart
  of the transformer, attention has a computational complexity of O(N^2),
  compared to O(N) for linear transformations. When handling large sequence
  lengths, attention becomes the primary time-consuming component. Although
  quantization has proven to be an effective method for accelerating model
  inference, existing quantization methods primarily focus on optimizing the
  linear layer. In response, we first analyze the feasibility of quantization in
  attention detailedly. Following that, we propose SageAttention, a highly
  efficient and accurate quantization method for attention. The OPS (operations
  per second) of our approach outperforms FlashAttention2 and xformers by about
  2.1 times and 2.7 times, respectively. SageAttention also achieves superior
  accuracy performance over FlashAttention3. Comprehensive experiments confirm
  that our approach incurs almost no end-to-end metrics loss across diverse
  models, including those for large language processing, image generation, and
  video generation.
paper_ai_keywords: transformer architecture attention computational complexity quantization model inference linear layer SageAttention operations per second FlashAttention2 xformers FlashAttention3 end-to-end metrics loss large language processing image generation video generation