MixLLM: LLM Quantization with Global Mixed-precision between
  Output-features and Highly-efficient System Design
Quantization has become one of the most effective methodologies to compress
LLMs into smaller size. However, the existing quantization solutions still show
limitations of either non-negligible accuracy drop or system inefficiency. In
this paper, we make a comprehensive analysis of the general quantization
principles on their effect to the triangle of accuracy, memory consumption and
system efficiency. We propose MixLLM that explores the new optimization space
of mixed-precision quantization between output features based on the insight
that different output features matter differently in the model. MixLLM
identifies the output features with high salience in the global view rather
than within each single layer, effectively assigning the larger bit-width to
output features that need it most to achieve good accuracy with low memory
consumption. We present the sweet spot of quantization configuration of
algorithm-system co-design that leads to high accuracy and system efficiency.
To address the system challenge, we design the two-step dequantization to make
use of the int8 Tensor Core easily and fast data type conversion to reduce
dequantization overhead significantly, and present the software pipeline to
overlap the memory access, dequantization and the MatMul to the best. Extensive
experiments show that with only 10% more bits, the PPL increasement can be
reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, while on
average MMLU-Pro improves by 0.93 over the SOTA of three popular models. In
addition to its superior accuracy, MixLLM also achieves state-of-the-art system
efficiency.
676827d8bc5af30a79857669
Zhen Zheng
claimed_verified
2024-12-23T11:10:47.069Z
65373b2c89dd48faca859d02
https://cdn-avatars.huggingface.co/v1/production/uploads/65373b2c89dd48faca859d02/42HukqnvMykvaoTxnQJjk.jpeg
Zhen Zheng
user
JamesTheZ
676827d8bc5af30a7985766a
Xiaonan Song
admin_assigned
2024-12-23T11:34:23.206Z
66839c29d88415938469b483
/avatars/38af0856922ea53f607a7158e640923c.svg
Xiaonan Song
user
xiaonans
676827d8bc5af30a7985766b
Chuanjie Liu
admin_assigned
2024-12-23T11:34:28.717Z
6528b76d0b44378dcc54ec60
/avatars/0943b4c7def02c1751e163aa758ccec5.svg
Chuanjie Liu
user
chuanjieliu
MixLLM: LLM Quantization with Global Mixed-precision between
  Output-features and Highly-efficient System Design
Quantization has become one of the most effective methodologies to compress
LLMs into smaller size. However, the existing quantization solutions still show
limitations of either non-negligible accuracy drop or system inefficiency. In
this paper, we make a comprehensive analysis of the general quantization
principles on their effect to the triangle of accuracy, memory consumption and
system efficiency. We propose MixLLM that explores the new optimization space
of mixed-precision quantization between output features based on the insight
that different output features matter differently in the model. MixLLM
identifies the output features with high salience in the global view rather
than within each single layer, effectively assigning the larger bit-width to
output features that need it most to achieve good accuracy with low memory
consumption. We present the sweet spot of quantization configuration of
algorithm-system co-design that leads to high accuracy and system efficiency.
To address the system challenge, we design the two-step dequantization to make
use of the int8 Tensor Core easily and fast data type conversion to reduce
dequantization overhead significantly, and present the software pipeline to
overlap the memory access, dequantization and the MatMul to the best. Extensive
experiments show that with only 10% more bits, the PPL increasement can be
reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, while on
average MMLU-Pro improves by 0.93 over the SOTA of three popular models. In
addition to its superior accuracy, MixLLM also achieves state-of-the-art system
efficiency.
MixLLM, a mixed-precision quantization technique, enhances the accuracy and efficiency of large language models by optimizing output feature quantization and system-level dequantization processes.
quantization mixed-precision quantization output features salience bit-width dequantization Tensor Core MatMul PPL MMLU-Pro system efficiency algorithm-system co-design