Layer-Condensed KV Cache for Efficient Inference of Large Language Models
<|reference_start|>Layer-Condensed KV Cache for Efficient Inference of Large Language Models: Huge memory consumption has been a major bottleneck for deploying high-throughput large language models in real-world applications. In addition to the large number of parameters, the key-value (KV) cache for the attention mechanism in the transformer architecture consumes a significant amount of memory, especially when the number of layers is large for deep language models. In this paper, we propose a novel method that only computes and caches the KVs of a small number of layers, thus significantly saving memory consumption and improving inference throughput. Our experiments on large language models show that our method achieves up to 26$\times$ higher throughput than standard transformers and competitive performance in language modeling and downstream tasks. In addition, our method is orthogonal to existing transformer memory-saving techniques, so it is straightforward to integrate them with our model, achieving further improvement in inference efficiency. Our code is available at https://github.com/whyNLP/LCKV.<|reference_end|>
@article{wu2024layer-condensed,
    title={Layer-Condensed KV Cache for Efficient Inference of Large Language
  Models},
    author={Haoyi Wu, Kewei Tu},
    journal={arXiv preprint arXiv:2405.10637},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10637},
    primaryClass={cs.CL}
}