title: TEQ: Trainable Equivalent Transformation for Quantization of LLMs
summary: As large language models (LLMs) become more prevalent, there is a growing
  need for new and improved quantization methods that can meet the
  computationalast layer demands of these modern architectures while maintaining
  the accuracy. In this paper, we present TEQ, a trainable equivalent
  transformation that preserves the FP32 precision of the model output while
  taking advantage of low-precision quantization, especially 3 and 4 bits
  weight-only quantization. The training process is lightweight, requiring only
  1K steps and fewer than 0.1 percent of the original model's trainable
  parameters. Furthermore, the transformation does not add any computational
  overhead during inference. Our results are on-par with the state-of-the-art
  (SOTA) methods on typical LLMs. Our approach can be combined with other methods
  to achieve even better performance. The code is available at
  https://github.com/intel/neural-compressor.
paper_authors._id: 652f4ee1da91a2e197b61c70
paper_authors.name: Wenhua Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:07:49.613Z
paper_authors._id: 634772780a9fa1ac49994510
paper_authors.avatarUrl: /avatars/2e88e32ac0d45a1c624026e497eb00b3.svg
paper_authors.fullname: wenhua cheng
paper_authors.type: user
paper_authors.user: wenhuach
paper_authors._id: 652f4ee1da91a2e197b61c71
paper_authors.name: Yiyang Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:08:23.832Z
paper_authors._id: 634e0e3b7521b97439520abe
paper_authors.avatarUrl: /avatars/d3aac6582dc4862ed640eff2ddb8b3c3.svg
paper_authors.fullname: Yiyang Cai
paper_authors.type: user
paper_authors.user: frankcai
paper_authors._id: 652f4ee1da91a2e197b61c72
paper_authors.name: Kaokao Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:08:41.064Z
paper_authors._id: 630881ff6fb2ea4413f16390
paper_authors.avatarUrl: /avatars/8cdf266604b24bc9ae599aa2def8debd.svg
paper_authors.fullname: lvkaokao
paper_authors.type: user
paper_authors.user: lvkaokao
paper_authors._id: 652f4ee1da91a2e197b61c73
paper_authors.name: Haihao Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:08:48.606Z
paper_authors._id: 60ac3318e3de7c7440abb850
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
paper_authors.fullname: Haihao Shen
paper_authors.type: user
paper_authors.user: Haihao
paper_title: TEQ: Trainable Equivalent Transformation for Quantization of LLMs
paper_summary: As large language models (LLMs) become more prevalent, there is a growing
  need for new and improved quantization methods that can meet the
  computationalast layer demands of these modern architectures while maintaining
  the accuracy. In this paper, we present TEQ, a trainable equivalent
  transformation that preserves the FP32 precision of the model output while
  taking advantage of low-precision quantization, especially 3 and 4 bits
  weight-only quantization. The training process is lightweight, requiring only
  1K steps and fewer than 0.1 percent of the original model's trainable
  parameters. Furthermore, the transformation does not add any computational
  overhead during inference. Our results are on-par with the state-of-the-art
  (SOTA) methods on typical LLMs. Our approach can be combined with other methods
  to achieve even better performance. The code is available at
  https://github.com/intel/neural-compressor.
paper_ai_summary: TEQ is a trainable transformation for low-precision quantization in large language models that maintains model accuracy with minimal computational overhead during inference.
paper_ai_keywords: LLMs quantization TEQ FP32 precision weight-only quantization inference