title: Training Transformers with 4-bit Integers
summary: Quantizing the activation, weight, and gradient to 4-bit is promising to
  accelerate neural network training. However, existing 4-bit training methods
  require custom numerical formats which are not supported by contemporary
  hardware. In this work, we propose a training method for transformers with all
  matrix multiplications implemented with the INT4 arithmetic. Training with an
  ultra-low INT4 precision is challenging. To achieve this, we carefully analyze
  the specific structures of activation and gradients in transformers to propose
  dedicated quantizers for them. For forward propagation, we identify the
  challenge of outliers and propose a Hadamard quantizer to suppress the
  outliers. For backpropagation, we leverage the structural sparsity of gradients
  by proposing bit splitting and leverage score sampling techniques to quantize
  gradients accurately. Our algorithm achieves competitive accuracy on a wide
  range of tasks including natural language understanding, machine translation,
  and image classification. Unlike previous 4-bit training methods, our algorithm
  can be implemented on the current generation of GPUs. Our prototypical linear
  operator implementation is up to 2.2 times faster than the FP16 counterparts
  and speeds up the training by up to 35.1%.
paper_authors._id: 6493c88bbc32063c9e46929b
paper_authors.name: Haocheng Xi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T07:31:51.474Z
paper_authors._id: 6452a2cbc895e6437dc567e2
paper_authors.avatarUrl: /avatars/f488ebe2fbece0e8ef4e9d6c95898f3a.svg
paper_authors.fullname: Haocheng Xi
paper_authors.type: user
paper_authors.user: Xihc20
paper_authors._id: 6493c88bbc32063c9e46929c
paper_authors.name: Changhao Li
paper_authors._id: 6493c88bbc32063c9e46929d
paper_authors.name: Jianfei Chen
paper_authors._id: 6493c88bbc32063c9e46929e
paper_authors.name: Jun Zhu
paper_title: Training Transformers with 4-bit Integers
paper_summary: Quantizing the activation, weight, and gradient to 4-bit is promising to
  accelerate neural network training. However, existing 4-bit training methods
  require custom numerical formats which are not supported by contemporary
  hardware. In this work, we propose a training method for transformers with all
  matrix multiplications implemented with the INT4 arithmetic. Training with an
  ultra-low INT4 precision is challenging. To achieve this, we carefully analyze
  the specific structures of activation and gradients in transformers to propose
  dedicated quantizers for them. For forward propagation, we identify the
  challenge of outliers and propose a Hadamard quantizer to suppress the
  outliers. For backpropagation, we leverage the structural sparsity of gradients
  by proposing bit splitting and leverage score sampling techniques to quantize
  gradients accurately. Our algorithm achieves competitive accuracy on a wide
  range of tasks including natural language understanding, machine translation,
  and image classification. Unlike previous 4-bit training methods, our algorithm
  can be implemented on the current generation of GPUs. Our prototypical linear
  operator implementation is up to 2.2 times faster than the FP16 counterparts
  and speeds up the training by up to 35.1%.
paper_ai_summary: A novel method for training transformers with 4-bit quantization achieves competitive accuracy and accelerates training on current GPUs.
paper_ai_keywords: activation quantization weight quantization gradient quantization INT4 arithmetic Hadamard quantizer bit splitting score sampling structural sparsity natural language understanding machine translation image classification FP16 counterparts