Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers
<|reference_start|>Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers: Non-linear functions are prevalent in Transformers and their lightweight variants, incurring substantial and frequently underestimated hardware costs. Previous state-of-the-art works optimize these operations by piece-wise linear approximation and store the parameters in look-up tables (LUT), but most of them require unfriendly high-precision arithmetics such as FP/INT 32 and lack consideration of integer-only INT quantization. This paper proposed a genetic LUT-Approximation algorithm namely GQA-LUT that can automatically determine the parameters with quantization awareness. The results demonstrate that GQA-LUT achieves negligible degradation on the challenging semantic segmentation task for both vanilla and linear Transformer models. Besides, proposed GQA-LUT enables the employment of INT8-based LUT-Approximation that achieves an area savings of 81.3~81.7% and a power reduction of 79.3~80.2% compared to the high-precision FP/INT 32 alternatives. Code is available at https:// github.com/PingchengDong/GQA-LUT.<|reference_end|>
@article{dong2024genetic,
    title={Genetic Quantization-Aware Approximation for Non-Linear Operations in
  Transformers},
    author={Pingcheng Dong, Yonghao Tan, Dong Zhang, Tianwei Ni, Xuejiao Liu, Yu
  Liu, Peng Luo, Luhong Liang, Shih-Yang Liu, Xijie Huang, Huaiyu Zhu, Yun Pan,
  Fengwei An, Kwang-Ting Cheng},
    journal={arXiv preprint arXiv:2403.19591},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.19591},
    primaryClass={cs.LG cs.AR cs.NE}
}