Layer-Condensed KV Cache for Efficient Inference of Large Language
  Models
Huge memory consumption has been a major bottleneck for deploying
high-throughput large language models in real-world applications. In addition
to the large number of parameters, the key-value (KV) cache for the attention
mechanism in the transformer architecture consumes a significant amount of
memory, especially when the number of layers is large for deep language models.
In this paper, we propose a novel method that only computes and caches the KVs
of a small number of layers, thus significantly saving memory consumption and
improving inference throughput. Our experiments on large language models show
that our method achieves up to 26times higher throughput than standard
transformers and competitive performance in language modeling and downstream
tasks. In addition, our method is orthogonal to existing transformer
memory-saving techniques, so it is straightforward to integrate them with our
model, achieving further improvement in inference efficiency. Our code is
available at https://github.com/whyNLP/LCKV.
664b1efef7e9d961cf850a77
Haoyi Wu
claimed_verified
2024-05-21T09:39:44.972Z
6469f91531fbfc5df81b9154
/avatars/20658b7341f181dbe702b4665d08780d.svg
whywhy
user
whynlp
664b1efef7e9d961cf850a78
Kewei Tu
Layer-Condensed KV Cache for Efficient Inference of Large Language
  Models
Huge memory consumption has been a major bottleneck for deploying
high-throughput large language models in real-world applications. In addition
to the large number of parameters, the key-value (KV) cache for the attention
mechanism in the transformer architecture consumes a significant amount of
memory, especially when the number of layers is large for deep language models.
In this paper, we propose a novel method that only computes and caches the KVs
of a small number of layers, thus significantly saving memory consumption and
improving inference throughput. Our experiments on large language models show
that our method achieves up to 26times higher throughput than standard
transformers and competitive performance in language modeling and downstream
tasks. In addition, our method is orthogonal to existing transformer
memory-saving techniques, so it is straightforward to integrate them with our
model, achieving further improvement in inference efficiency. Our code is
available at https://github.com/whyNLP/LCKV.
A method reducing key-value cache memory in transformer architectures improves inference throughput without sacrificing performance in language modeling tasks.
key-value cache attention mechanism transformer architecture inference throughput