title: Layer-Condensed KV Cache for Efficient Inference of Large Language
    Models
summary: Huge memory consumption has been a major bottleneck for deploying
  high-throughput large language models in real-world applications. In addition
  to the large number of parameters, the key-value (KV) cache for the attention
  mechanism in the transformer architecture consumes a significant amount of
  memory, especially when the number of layers is large for deep language models.
  In this paper, we propose a novel method that only computes and caches the KVs
  of a small number of layers, thus significantly saving memory consumption and
  improving inference throughput. Our experiments on large language models show
  that our method achieves up to 26times higher throughput than standard
  transformers and competitive performance in language modeling and downstream
  tasks. In addition, our method is orthogonal to existing transformer
  memory-saving techniques, so it is straightforward to integrate them with our
  model, achieving further improvement in inference efficiency. Our code is
  available at https://github.com/whyNLP/LCKV.
paper_authors._id: 664b1efef7e9d961cf850a77
paper_authors.name: Haoyi Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-21T09:39:44.972Z
paper_authors._id: 6469f91531fbfc5df81b9154
paper_authors.avatarUrl: /avatars/20658b7341f181dbe702b4665d08780d.svg
paper_authors.fullname: whywhy
paper_authors.type: user
paper_authors.user: whynlp
paper_authors._id: 664b1efef7e9d961cf850a78
paper_authors.name: Kewei Tu
paper_title: Layer-Condensed KV Cache for Efficient Inference of Large Language
    Models
paper_summary: Huge memory consumption has been a major bottleneck for deploying
  high-throughput large language models in real-world applications. In addition
  to the large number of parameters, the key-value (KV) cache for the attention
  mechanism in the transformer architecture consumes a significant amount of
  memory, especially when the number of layers is large for deep language models.
  In this paper, we propose a novel method that only computes and caches the KVs
  of a small number of layers, thus significantly saving memory consumption and
  improving inference throughput. Our experiments on large language models show
  that our method achieves up to 26times higher throughput than standard
  transformers and competitive performance in language modeling and downstream
  tasks. In addition, our method is orthogonal to existing transformer
  memory-saving techniques, so it is straightforward to integrate them with our
  model, achieving further improvement in inference efficiency. Our code is
  available at https://github.com/whyNLP/LCKV.
paper_ai_keywords: key-value (KV) cache attention mechanism transformer architecture inference throughput language modeling downstream tasks inference efficiency