PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers
  in LLMs
Quantization is essential for deploying Large Language Models (LLMs) by
enhancing memory efficiency and inference speed. Existing methods for
activation quantization mainly address channel-wise outliers, often neglecting
token-wise outliers, leading to reliance on costly per-token dynamic
quantization. To address this, we introduce PrefixQuant, a novel technique that
isolates outlier tokens offline without re-training. Specifically, PrefixQuant
identifies high-frequency outlier tokens and prefixes them in the KV cache,
preventing the generation of outlier tokens during inference and simplifying
quantization. To our knowledge, PrefixQuant is the first to enable efficient
per-tensor static quantization to outperform expensive per-token dynamic
quantization. For instance, in W4A4KV4 (4- bit weight, 4-bit activation, and
4-bit KV cache) Llama-3-8B, PrefixQuant with per-tensor static quantization
achieves a 7.43 WikiText2 perplexity and 71.08% average accuracy on 5
common-sense reasoning tasks, outperforming previous per-token dynamic
quantization methods like QuaRot with 0.98 perplexity improvement and +5.98
points accuracy. Additionally, the inference speed of W4A4 quantized models
using PrefixQuant is 1.60x to 2.81x faster than FP16 models and exceeds QuaRot
models by 1.2x to 1.3x. Our code is available at
https://github.com/ChenMnZ/PrefixQuant.
67089018f5c3ffa260bc592b
Mengzhao Chen
admin_assigned
2024-10-11T10:13:34.902Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
67089018f5c3ffa260bc592c
Yi Liu
claimed_verified
2024-10-13T20:16:14.960Z
670611a919b7acaedc21f3d5
/avatars/8b051543b22fba98c2eb6007b3ceec92.svg
Yi Liu
user
Yi30
67089018f5c3ffa260bc592d
Jiahao Wang
67089018f5c3ffa260bc592e
Yi Bin
67089018f5c3ffa260bc592f
Wenqi Shao
admin_assigned
2024-10-11T10:14:31.756Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
67089018f5c3ffa260bc5930
Ping Luo
PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers
  in LLMs
Quantization is essential for deploying Large Language Models (LLMs) by
enhancing memory efficiency and inference speed. Existing methods for
activation quantization mainly address channel-wise outliers, often neglecting
token-wise outliers, leading to reliance on costly per-token dynamic
quantization. To address this, we introduce PrefixQuant, a novel technique that
isolates outlier tokens offline without re-training. Specifically, PrefixQuant
identifies high-frequency outlier tokens and prefixes them in the KV cache,
preventing the generation of outlier tokens during inference and simplifying
quantization. To our knowledge, PrefixQuant is the first to enable efficient
per-tensor static quantization to outperform expensive per-token dynamic
quantization. For instance, in W4A4KV4 (4- bit weight, 4-bit activation, and
4-bit KV cache) Llama-3-8B, PrefixQuant with per-tensor static quantization
achieves a 7.43 WikiText2 perplexity and 71.08% average accuracy on 5
common-sense reasoning tasks, outperforming previous per-token dynamic
quantization methods like QuaRot with 0.98 perplexity improvement and +5.98
points accuracy. Additionally, the inference speed of W4A4 quantized models
using PrefixQuant is 1.60x to 2.81x faster than FP16 models and exceeds QuaRot
models by 1.2x to 1.3x. Our code is available at
https://github.com/ChenMnZ/PrefixQuant.
PrefixQuant improves quantization efficiency in LLMs by isolating outlier tokens, enabling static quantization to outperform dynamic quantization in both speed and accuracy.
quantization Large Language Models (LLMs) memory efficiency inference speed activation quantization channel-wise outliers token-wise outliers PrefixQuant KV cache per-tensor static quantization per-token dynamic quantization W4A4KV4 Llama-3-8B WikiText2 perplexity common-sense reasoning tasks QuaRot