TEQ: Trainable Equivalent Transformation for Quantization of LLMs
As large language models (LLMs) become more prevalent, there is a growing
need for new and improved quantization methods that can meet the
computationalast layer demands of these modern architectures while maintaining
the accuracy. In this paper, we present TEQ, a trainable equivalent
transformation that preserves the FP32 precision of the model output while
taking advantage of low-precision quantization, especially 3 and 4 bits
weight-only quantization. The training process is lightweight, requiring only
1K steps and fewer than 0.1 percent of the original model's trainable
parameters. Furthermore, the transformation does not add any computational
overhead during inference. Our results are on-par with the state-of-the-art
(SOTA) methods on typical LLMs. Our approach can be combined with other methods
to achieve even better performance. The code is available at
https://github.com/intel/neural-compressor.
652f4ee1da91a2e197b61c70
Wenhua Cheng
admin_assigned
2023-10-18T09:07:49.613Z
634772780a9fa1ac49994510
/avatars/2e88e32ac0d45a1c624026e497eb00b3.svg
wenhua cheng
user
wenhuach
652f4ee1da91a2e197b61c71
Yiyang Cai
admin_assigned
2023-10-18T09:08:23.832Z
634e0e3b7521b97439520abe
/avatars/d3aac6582dc4862ed640eff2ddb8b3c3.svg
Yiyang Cai
user
frankcai
652f4ee1da91a2e197b61c72
Kaokao Lv
admin_assigned
2023-10-18T09:08:41.064Z
630881ff6fb2ea4413f16390
/avatars/8cdf266604b24bc9ae599aa2def8debd.svg
lvkaokao
user
lvkaokao
652f4ee1da91a2e197b61c73
Haihao Shen
admin_assigned
2023-10-18T09:08:48.606Z
60ac3318e3de7c7440abb850
https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
Haihao Shen
user
Haihao
TEQ: Trainable Equivalent Transformation for Quantization of LLMs
As large language models (LLMs) become more prevalent, there is a growing
need for new and improved quantization methods that can meet the
computationalast layer demands of these modern architectures while maintaining
the accuracy. In this paper, we present TEQ, a trainable equivalent
transformation that preserves the FP32 precision of the model output while
taking advantage of low-precision quantization, especially 3 and 4 bits
weight-only quantization. The training process is lightweight, requiring only
1K steps and fewer than 0.1 percent of the original model's trainable
parameters. Furthermore, the transformation does not add any computational
overhead during inference. Our results are on-par with the state-of-the-art
(SOTA) methods on typical LLMs. Our approach can be combined with other methods
to achieve even better performance. The code is available at
https://github.com/intel/neural-compressor.
TEQ is a trainable transformation for low-precision quantization in large language models that maintains model accuracy with minimal computational overhead during inference.
LLMs quantization TEQ FP32 precision weight-only quantization inference