Attention-aware Post-training Quantization without Backpropagation
<|reference_start|>Attention-aware Post-training Quantization without Backpropagation: Quantization is a promising solution for deploying large-scale language models (LLMs) on resource-constrained devices. Existing quantization approaches, however, rely on gradient-based optimization, regardless of it being post-training quantization (PTQ) or quantization-aware training (QAT), which becomes problematic for hyper-scale LLMs with billions of parameters. This overhead can be alleviated via recently proposed backpropagation-free PTQ methods; however, their performance is somewhat limited by their lack of consideration of inter-layer dependencies. In this paper, we thus propose a novel PTQ algorithm that considers inter-layer dependencies without relying on backpropagation. The fundamental concept involved is the development of attention-aware Hessian matrices, which facilitates the consideration of inter-layer dependencies within the attention module. Extensive experiments demonstrate that the proposed algorithm significantly outperforms conventional PTQ methods, particularly for low bit-widths.<|reference_end|>
@article{kim2024attention-aware,
    title={Attention-aware Post-training Quantization without Backpropagation},
    author={Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim,
  Yongkweon Jeon},
    journal={arXiv preprint arXiv:2406.13474},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.13474},
    primaryClass={cs.LG cs.AI}
}