WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
<|reference_start|>WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More: Large Language Models (LLMs) face significant deployment challenges due to their substantial memory requirements and the computational demands of auto-regressive text generation process. This paper addresses these challenges by focusing on the quantization of LLMs, a technique that reduces memory consumption by converting model parameters and activations into low-bit integers. We critically analyze the existing quantization approaches, identifying their limitations in balancing the accuracy and efficiency of the quantized LLMs. To advance beyond these limitations, we propose WKVQuant, a PTQ framework especially designed for quantizing weights and the key/value (KV) cache of LLMs. Specifically, we incorporates past-only quantization to improve the computation of attention. Additionally, we introduce two-dimensional quantization strategy to handle the distribution of KV cache, along with a cross-block reconstruction regularization for parameter optimization. Experiments show that WKVQuant achieves almost comparable memory savings to weight-activation quantization, while also approaching the performance of weight-only quantization.<|reference_end|>
@article{yue2024wkvquant:,
    title={WKVQuant: Quantizing Weight and Key/Value Cache for Large Language
  Models Gains More},
    author={Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu,
  Liqiang Nie},
    journal={arXiv preprint arXiv:2402.12065},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.12065},
    primaryClass={cs.LG cs.AI cs.CL}
}