title: Scaling Law for Quantization-Aware Training
summary: Large language models (LLMs) demand substantial computational and memory
  resources, creating deployment challenges. Quantization-aware training (QAT)
  addresses these challenges by reducing model precision while maintaining
  performance. However, the scaling behavior of QAT, especially at 4-bit
  precision (W4A4), is not well understood. Existing QAT scaling laws often
  ignore key factors such as the number of training tokens and quantization
  granularity, which limits their applicability. This paper proposes a unified
  scaling law for QAT that models quantization error as a function of model size,
  training data volume, and quantization group size. Through 268 QAT experiments,
  we show that quantization error decreases as model size increases, but rises
  with more training tokens and coarser quantization granularity. To identify the
  sources of W4A4 quantization error, we decompose it into weight and activation
  components. Both components follow the overall trend of W4A4 quantization
  error, but with different sensitivities. Specifically, weight quantization
  error increases more rapidly with more training tokens. Further analysis shows
  that the activation quantization error in the FC2 layer, caused by outliers, is
  the primary bottleneck of W4A4 QAT quantization error. By applying
  mixed-precision quantization to address this bottleneck, we demonstrate that
  weight and activation quantization errors can converge to similar levels.
  Additionally, with more training data, weight quantization error eventually
  exceeds activation quantization error, suggesting that reducing weight
  quantization error is also important in such scenarios. These findings offer
  key insights for improving QAT research and development.
paper_authors._id: 682e887e866a44f6a81409b1
paper_authors.name: Mengzhao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:29.873Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 682e887e866a44f6a81409b2
paper_authors.name: Chaoyi Zhang
paper_authors._id: 682e887e866a44f6a81409b3
paper_authors.name: Jing Liu
paper_authors._id: 682e887e866a44f6a81409b4
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:22:15.288Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 682e887e866a44f6a81409b5
paper_authors.name: Zeyue Xue
paper_authors._id: 682e887e866a44f6a81409b6
paper_authors.name: Zhiheng Liu
paper_authors._id: 682e887e866a44f6a81409b7
paper_authors.name: Yunshui Li
paper_authors._id: 682e887e866a44f6a81409b8
paper_authors.name: Jin Ma
paper_authors._id: 682e887e866a44f6a81409b9
paper_authors.name: Jie Huang
paper_authors._id: 682e887e866a44f6a81409ba
paper_authors.name: Xun Zhou
paper_authors._id: 682e887e866a44f6a81409bb
paper_authors.name: Ping Luo
paper_title: Scaling Law for Quantization-Aware Training
paper_summary: Large language models (LLMs) demand substantial computational and memory
  resources, creating deployment challenges. Quantization-aware training (QAT)
  addresses these challenges by reducing model precision while maintaining
  performance. However, the scaling behavior of QAT, especially at 4-bit
  precision (W4A4), is not well understood. Existing QAT scaling laws often
  ignore key factors such as the number of training tokens and quantization
  granularity, which limits their applicability. This paper proposes a unified
  scaling law for QAT that models quantization error as a function of model size,
  training data volume, and quantization group size. Through 268 QAT experiments,
  we show that quantization error decreases as model size increases, but rises
  with more training tokens and coarser quantization granularity. To identify the
  sources of W4A4 quantization error, we decompose it into weight and activation
  components. Both components follow the overall trend of W4A4 quantization
  error, but with different sensitivities. Specifically, weight quantization
  error increases more rapidly with more training tokens. Further analysis shows
  that the activation quantization error in the FC2 layer, caused by outliers, is
  the primary bottleneck of W4A4 QAT quantization error. By applying
  mixed-precision quantization to address this bottleneck, we demonstrate that
  weight and activation quantization errors can converge to similar levels.
  Additionally, with more training data, weight quantization error eventually
  exceeds activation quantization error, suggesting that reducing weight
  quantization error is also important in such scenarios. These findings offer
  key insights for improving QAT research and development.
paper_ai_summary: A unified scaling law for quantization-aware training (QAT) identifies key factors affecting quantization error, leading to improvements through mixed-precision quantization.
paper_ai_keywords: quantization-aware training QAT quantization error model size training tokens quantization granularity weight quantization activation quantization mixed-precision quantization FC2 layer