LQER: Low-Rank Quantization Error Reconstruction for LLMs
<|reference_start|>LQER: Low-Rank Quantization Error Reconstruction for LLMs: Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-base iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36$\times$ fewer hardware resources than the leading state-of-the-art method. We open-source our framework at https://github.com/ChengZhang-98/lqer<|reference_end|>
@article{zhang2024lqer:,
    title={LQER: Low-Rank Quantization Error Reconstruction for LLMs},
    author={Cheng Zhang, Jianyi Cheng, George A. Constantinides, and Yiren Zhao},
    journal={arXiv preprint arXiv:2402.02446},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.02446},
    primaryClass={cs.LG cs.CL}
}