APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
<|reference_start|>APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models: Large Language Models (LLMs) have greatly advanced the natural language processing paradigm. However, the high computational load and huge model sizes pose a grand challenge for deployment on edge devices. To this end, we propose APTQ (Attention-aware Post-Training Mixed-Precision Quantization) for LLMs, which considers not only the second-order information of each layer's weights, but also, for the first time, the nonlinear effect of attention outputs on the entire model. We leverage the Hessian trace as a sensitivity metric for mixed-precision quantization, ensuring an informed precision reduction that retains model performance. Experiments show APTQ surpasses previous quantization methods, achieving an average of 4 bit width a 5.22 perplexity nearly equivalent to full precision in the C4 dataset. In addition, APTQ attains state-of-the-art zero-shot accuracy of 68.24\% and 70.48\% at an average bitwidth of 3.8 in LLaMa-7B and LLaMa-13B, respectively, demonstrating its effectiveness to produce high-quality quantized LLMs.<|reference_end|>
@article{guan2024aptq:,
    title={APTQ: Attention-aware Post-Training Mixed-Precision Quantization for
  Large Language Models},
    author={Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong and Hao Yu},
    journal={arXiv preprint arXiv:2402.14866},
    year={2024},
    doi={10.1145/3649329.3658498},
    archivePrefix={arXiv},
    eprint={2402.14866},
    primaryClass={cs.LG cs.AI cs.CL}
}