A method of using RSVD in residual calculation of LowBit GEMM
<|reference_start|>A method of using RSVD in residual calculation of LowBit GEMM: The advancements of hardware technology in recent years has brought many possibilities for low-precision applications. However, the use of low precision can introduce significant computational errors, posing a considerable challenge to maintaining the computational accuracy.   We propose low-rank residuals quantized matrix multiplication(LRQMM) method which introduces low-rank approximation in residual compensation for dense low precision quantization matrix multiplication. It can bring several times accuracy improvement with only BLAS-2 level extra time overhead. Moreover, LRQMM is a completely data-free quantization method that does not require additional data for pre-training. And it only works with low precision GEMM operator, which is easy to couple with other methods.   Through experimentation, LRQMM can reduce the error of direct quantized matrix multiplication by 1~2 orders of magnitude, when dealing with larger matrix sizes, the computational speed is only reduced by approximately 20\%. In deep learning networks, LRQMM-4bit achieves 61.8% ImageNet Top-1 accuracy in Resnet-50, while the Direct Quant accuracy is only 8.3%.<|reference_end|>
@article{gu2024a,
    title={A method of using RSVD in residual calculation of LowBit GEMM},
    author={Hongyaoxing Gu},
    journal={arXiv preprint arXiv:2409.18772},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.18772},
    primaryClass={cs.MS cs.LG}
}