title: PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers
    in LLMs
summary: Quantization is essential for deploying Large Language Models (LLMs) by
  enhancing memory efficiency and inference speed. Existing methods for
  activation quantization mainly address channel-wise outliers, often neglecting
  token-wise outliers, leading to reliance on costly per-token dynamic
  quantization. To address this, we introduce PrefixQuant, a novel technique that
  isolates outlier tokens offline without re-training. Specifically, PrefixQuant
  identifies high-frequency outlier tokens and prefixes them in the KV cache,
  preventing the generation of outlier tokens during inference and simplifying
  quantization. To our knowledge, PrefixQuant is the first to enable efficient
  per-tensor static quantization to outperform expensive per-token dynamic
  quantization. For instance, in W4A4KV4 (4- bit weight, 4-bit activation, and
  4-bit KV cache) Llama-3-8B, PrefixQuant with per-tensor static quantization
  achieves a 7.43 WikiText2 perplexity and 71.08% average accuracy on 5
  common-sense reasoning tasks, outperforming previous per-token dynamic
  quantization methods like QuaRot with 0.98 perplexity improvement and +5.98
  points accuracy. Additionally, the inference speed of W4A4 quantized models
  using PrefixQuant is 1.60x to 2.81x faster than FP16 models and exceeds QuaRot
  models by 1.2x to 1.3x. Our code is available at
  https://github.com/ChenMnZ/PrefixQuant.
paper_authors._id: 67089018f5c3ffa260bc592b
paper_authors.name: Mengzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:13:34.902Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 67089018f5c3ffa260bc592c
paper_authors.name: Yi Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:16:14.960Z
paper_authors._id: 670611a919b7acaedc21f3d5
paper_authors.avatarUrl: /avatars/8b051543b22fba98c2eb6007b3ceec92.svg
paper_authors.fullname: Yi Liu
paper_authors.type: user
paper_authors.user: Yi30
paper_authors._id: 67089018f5c3ffa260bc592d
paper_authors.name: Jiahao Wang
paper_authors._id: 67089018f5c3ffa260bc592e
paper_authors.name: Yi Bin
paper_authors._id: 67089018f5c3ffa260bc592f
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:14:31.756Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 67089018f5c3ffa260bc5930
paper_authors.name: Ping Luo
paper_title: PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers
    in LLMs
paper_summary: Quantization is essential for deploying Large Language Models (LLMs) by
  enhancing memory efficiency and inference speed. Existing methods for
  activation quantization mainly address channel-wise outliers, often neglecting
  token-wise outliers, leading to reliance on costly per-token dynamic
  quantization. To address this, we introduce PrefixQuant, a novel technique that
  isolates outlier tokens offline without re-training. Specifically, PrefixQuant
  identifies high-frequency outlier tokens and prefixes them in the KV cache,
  preventing the generation of outlier tokens during inference and simplifying
  quantization. To our knowledge, PrefixQuant is the first to enable efficient
  per-tensor static quantization to outperform expensive per-token dynamic
  quantization. For instance, in W4A4KV4 (4- bit weight, 4-bit activation, and
  4-bit KV cache) Llama-3-8B, PrefixQuant with per-tensor static quantization
  achieves a 7.43 WikiText2 perplexity and 71.08% average accuracy on 5
  common-sense reasoning tasks, outperforming previous per-token dynamic
  quantization methods like QuaRot with 0.98 perplexity improvement and +5.98
  points accuracy. Additionally, the inference speed of W4A4 quantized models
  using PrefixQuant is 1.60x to 2.81x faster than FP16 models and exceeds QuaRot
  models by 1.2x to 1.3x. Our code is available at
  https://github.com/ChenMnZ/PrefixQuant.
paper_ai_summary: PrefixQuant improves quantization efficiency in LLMs by isolating outlier tokens, enabling static quantization to outperform dynamic quantization in both speed and accuracy.
paper_ai_keywords: quantization Large Language Models (LLMs) memory efficiency inference speed activation quantization channel-wise outliers token-wise outliers PrefixQuant KV cache per-tensor static quantization per-token dynamic quantization W4A4KV4 Llama-3-8B WikiText2 perplexity common-sense reasoning tasks QuaRot