Scaling Laws for Mixed quantization in Large Language Models
<|reference_start|>Scaling Laws for Mixed quantization in Large Language Models: Post-training quantization of Large Language Models (LLMs) has proven effective in reducing the computational requirements for running inference on these models. In this study, we focus on a straightforward question: When aiming for a specific accuracy or perplexity target for low-precision quantization, how many high-precision numbers or calculations are required to preserve as we scale LLMs to larger sizes? We first introduce a critical metric named the quantization ratio, which compares the number of parameters quantized to low-precision arithmetic against the total parameter count. Through extensive and carefully controlled experiments across different model families, arithmetic types, and quantization granularities (e.g. layer-wise, matmul-wise), we identify two central phenomenons. 1) The larger the models, the better they can preserve performance with an increased quantization ratio, as measured by perplexity in pre-training tasks or accuracy in downstream tasks. 2) The finer the granularity of mixed-precision quantization (e.g., matmul-wise), the more the model can increase the quantization ratio. We believe these observed phenomena offer valuable insights for future AI hardware design and the development of advanced Efficient AI algorithms.<|reference_end|>
@article{cao2024scaling,
    title={Scaling Laws for Mixed quantization in Large Language Models},
    author={Zeyu Cao, Cheng Zhang, Pedro Gimenes, Jianqiao Lu, Jianyi Cheng, Yiren
  Zhao},
    journal={arXiv preprint arXiv:2410.06722},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.06722},
    primaryClass={cs.CL cs.LG}
}