title: Quantizable Transformers: Removing Outliers by Helping Attention Heads
    Do Nothing
summary: Transformer models have been widely adopted in various domains over the last
  years, and especially large language models have advanced the field of AI
  significantly. Due to their size, the capability of these networks has
  increased tremendously, but this has come at the cost of a significant increase
  in necessary compute. Quantization is one of the most effective ways to reduce
  the computational time and memory consumption of neural networks. Many studies
  have shown, however, that modern transformer models tend to learn strong
  outliers in their activations, making them difficult to quantize. To retain
  acceptable performance, the existence of these outliers requires activations to
  be in higher bitwidth or the use of different numeric formats, extra
  fine-tuning, or other workarounds. We show that strong outliers are related to
  very specific behavior of attention heads that try to learn a "no-op" or just a
  partial update of the residual. To achieve the exact zeros needed in the
  attention matrix for a no-update, the input to the softmax is pushed to be
  larger and larger during training, causing outliers in other parts of the
  network. Based on these observations, we propose two simple (independent)
  modifications to the attention mechanism - clipped softmax and gated attention.
  We empirically show that models pre-trained using our methods learn
  significantly smaller outliers while maintaining and sometimes even improving
  the floating-point task performance. This enables us to quantize transformers
  to full INT8 quantization of the activations without any additional effort. We
  demonstrate the effectiveness of our methods on both language models (BERT,
  OPT) and vision transformers.
paper_authors._id: 6494f16de7ed8fa81c628049
paper_authors.name: Yelysei Bondarenko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T12:20:25.113Z
paper_authors._id: 649571182f09f73fc9c71340
paper_authors.avatarUrl: /avatars/5b51bbeac91b6a53712e61a30b670e67.svg
paper_authors.fullname: Bondarenko
paper_authors.type: user
paper_authors.user: Yelysei
paper_authors._id: 6494f16de7ed8fa81c62804a
paper_authors.name: Markus Nagel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T12:20:28.930Z
paper_authors._id: 649576b545b2cf0bf02319f1
paper_authors.avatarUrl: /avatars/a25b056ce3a354f21a8074d427ccbd2e.svg
paper_authors.fullname: Markus Nagel
paper_authors.type: user
paper_authors.user: markusnagel
paper_authors._id: 6494f16de7ed8fa81c62804b
paper_authors.name: Tijmen Blankevoort
paper_title: Quantizable Transformers: Removing Outliers by Helping Attention Heads
    Do Nothing
paper_summary: Transformer models have been widely adopted in various domains over the last
  years, and especially large language models have advanced the field of AI
  significantly. Due to their size, the capability of these networks has
  increased tremendously, but this has come at the cost of a significant increase
  in necessary compute. Quantization is one of the most effective ways to reduce
  the computational time and memory consumption of neural networks. Many studies
  have shown, however, that modern transformer models tend to learn strong
  outliers in their activations, making them difficult to quantize. To retain
  acceptable performance, the existence of these outliers requires activations to
  be in higher bitwidth or the use of different numeric formats, extra
  fine-tuning, or other workarounds. We show that strong outliers are related to
  very specific behavior of attention heads that try to learn a "no-op" or just a
  partial update of the residual. To achieve the exact zeros needed in the
  attention matrix for a no-update, the input to the softmax is pushed to be
  larger and larger during training, causing outliers in other parts of the
  network. Based on these observations, we propose two simple (independent)
  modifications to the attention mechanism - clipped softmax and gated attention.
  We empirically show that models pre-trained using our methods learn
  significantly smaller outliers while maintaining and sometimes even improving
  the floating-point task performance. This enables us to quantize transformers
  to full INT8 quantization of the activations without any additional effort. We
  demonstrate the effectiveness of our methods on both language models (BERT,
  OPT) and vision transformers.
paper_ai_keywords: Transformer models large language models quantization activations outliers attention heads "no-op" partial update residual softmax attention mechanism clipped softmax gated attention full INT8 quantization BERT OPT vision transformers