Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
<|reference_start|>Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers: With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyper-parameter tunings are required. As a cost-effective alternative, one-shot PTQ schemes have been proposed. Still, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a very important feature of Transformers. In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while considering cross-layer dependency to preserve the attention score. Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models.<|reference_end|>
@article{kim2024towards,
    title={Towards Next-Level Post-Training Quantization of Hyper-Scale
  Transformers},
    author={Junhan Kim, Chungman Lee, Eulrang Cho, Kyungphil Park, Ho-young Kim,
  Joonyoung Kim, Yongkweon Jeon},
    journal={arXiv preprint arXiv:2402.08958},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.08958},
    primaryClass={cs.LG cs.AI}
}