title: A Simple and Effective L_2 Norm-Based Strategy for KV Cache
    Compression
summary: The deployment of large language models (LLMs) is often hindered by the
  extensive memory requirements of the Key-Value (KV) cache, especially as
  context lengths increase. Existing approaches to reduce the KV cache size
  involve either fine-tuning the model to learn a compression strategy or
  leveraging attention scores to reduce the sequence length. We analyse the
  attention distributions in decoder-only Transformers-based models and observe
  that attention allocation patterns stay consistent across most layers.
  Surprisingly, we find a clear correlation between the L_2 and the attention
  scores over cached KV pairs, where a low L_2 of a key embedding usually leads
  to a high attention score during decoding. This finding indicates that the
  influence of a KV pair is potentially determined by the key embedding itself
  before being queried. Based on this observation, we compress the KV cache based
  on the L_2 of key embeddings. Our experimental results show that this simple
  strategy can reduce the KV cache size by 50% on language modelling and
  needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing
  accuracy.
paper_authors._id: 66716ba1d059806ae6e4ba43
paper_authors.name: Alessio Devoto
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:56:26.707Z
paper_authors._id: 62180288d05d07ebf7de5f31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
paper_authors.fullname: Alessio
paper_authors.type: user
paper_authors.user: alessiodevoto
paper_authors._id: 66716ba1d059806ae6e4ba44
paper_authors.name: Yu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T15:55:05.167Z
paper_authors._id: 654a500d9b8bd6406d431c0d
paper_authors.avatarUrl: /avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
paper_authors.fullname: Yu Zhao
paper_authors.type: user
paper_authors.user: yuzhaouoe
paper_authors._id: 66716ba1d059806ae6e4ba45
paper_authors.name: Simone Scardapane
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-21T15:20:24.561Z
paper_authors._id: 6321a810a97afe3c4c64457f
paper_authors.avatarUrl: /avatars/769bd3874163823733c817d8707e279f.svg
paper_authors.fullname: Simone Scardapane
paper_authors.type: user
paper_authors.user: sscardapane
paper_authors._id: 66716ba1d059806ae6e4ba46
paper_authors.name: Pasquale Minervini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T16:29:44.308Z
paper_authors._id: 61001311e043e15c13412d30
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
paper_authors.fullname: Pasquale Minervini
paper_authors.type: user
paper_authors.user: pminervini
paper_title: A Simple and Effective L_2 Norm-Based Strategy for KV Cache
    Compression
paper_summary: The deployment of large language models (LLMs) is often hindered by the
  extensive memory requirements of the Key-Value (KV) cache, especially as
  context lengths increase. Existing approaches to reduce the KV cache size
  involve either fine-tuning the model to learn a compression strategy or
  leveraging attention scores to reduce the sequence length. We analyse the
  attention distributions in decoder-only Transformers-based models and observe
  that attention allocation patterns stay consistent across most layers.
  Surprisingly, we find a clear correlation between the L_2 and the attention
  scores over cached KV pairs, where a low L_2 of a key embedding usually leads
  to a high attention score during decoding. This finding indicates that the
  influence of a KV pair is potentially determined by the key embedding itself
  before being queried. Based on this observation, we compress the KV cache based
  on the L_2 of key embeddings. Our experimental results show that this simple
  strategy can reduce the KV cache size by 50% on language modelling and
  needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing
  accuracy.
paper_ai_summary: By compressing the KV cache based on the $L_2$ norm of key embeddings, the sizes required for transformer models can be significantly reduced without sacrificing accuracy.
paper_ai_keywords: large language models key-value cache decoder-only Transformers attention distributions $L_2$ norm key embedding language modelling needle-in-a-haystack tasks passkey retrieval tasks