title: Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
summary: Autoregressive language models rely on a Key-Value (KV) Cache, which avoids
  re-computing past hidden states during generation, making it faster. As model
  sizes and context lengths grow, the KV Cache becomes a significant memory
  bottleneck, which calls for compression methods that limit its size during
  generation. In this paper, we discover surprising properties of Query (Q) and
  Key (K) vectors that allow us to efficiently approximate attention scores
  without computing the attention maps. We propose Q-Filters, a training-free KV
  Cache compression method that filters out less crucial Key-Value pairs based on
  a single context-agnostic projection. Contrarily to many alternatives,
  Q-Filters is compatible with FlashAttention, as it does not require direct
  access to attention weights. Experimental results in long-context settings
  demonstrate that Q-Filters is competitive with attention-based compression
  methods such as SnapKV in retrieval tasks while consistently outperforming
  efficient compression schemes such as Streaming-LLM in generation setups.
  Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task
  with a x32 compression level while reducing the generation perplexity drop by
  up to 65% in text generation compared to Streaming-LLM.
paper_authors._id: 67c87b12d62973a5d9f014b3
paper_authors.name: Nathan Godey
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-05T16:25:56.974Z
paper_authors._id: 61d2d6a5291690e1c7b4dd2d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666688353023-61d2d6a5291690e1c7b4dd2d.jpeg
paper_authors.fullname: Nathan Godey
paper_authors.type: user
paper_authors.user: nthngdy
paper_authors._id: 67c87b12d62973a5d9f014b4
paper_authors.name: Alessio Devoto
paper_authors._id: 67c87b12d62973a5d9f014b5
paper_authors.name: Yu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T13:34:20.584Z
paper_authors._id: 654a500d9b8bd6406d431c0d
paper_authors.avatarUrl: /avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
paper_authors.fullname: Yu Zhao
paper_authors.type: user
paper_authors.user: yuzhaouoe
paper_authors._id: 67c87b12d62973a5d9f014b6
paper_authors.name: Simone Scardapane
paper_authors._id: 67c87b12d62973a5d9f014b7
paper_authors.name: Pasquale Minervini
paper_authors._id: 67c87b12d62973a5d9f014b8
paper_authors.name: Éric de la Clergerie
paper_authors._id: 67c87b12d62973a5d9f014b9
paper_authors.name: Benoît Sagot
paper_title: Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
paper_summary: Autoregressive language models rely on a Key-Value (KV) Cache, which avoids
  re-computing past hidden states during generation, making it faster. As model
  sizes and context lengths grow, the KV Cache becomes a significant memory
  bottleneck, which calls for compression methods that limit its size during
  generation. In this paper, we discover surprising properties of Query (Q) and
  Key (K) vectors that allow us to efficiently approximate attention scores
  without computing the attention maps. We propose Q-Filters, a training-free KV
  Cache compression method that filters out less crucial Key-Value pairs based on
  a single context-agnostic projection. Contrarily to many alternatives,
  Q-Filters is compatible with FlashAttention, as it does not require direct
  access to attention weights. Experimental results in long-context settings
  demonstrate that Q-Filters is competitive with attention-based compression
  methods such as SnapKV in retrieval tasks while consistently outperforming
  efficient compression schemes such as Streaming-LLM in generation setups.
  Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task
  with a x32 compression level while reducing the generation perplexity drop by
  up to 65% in text generation compared to Streaming-LLM.
paper_ai_summary: Q-Filters is a KV Cache compression method using Q-Filters that efficiently reduces memory usage without compromising attention scores and text generation quality.
paper_ai_keywords: Key-Value Cache autoregressive language models Query vectors Key vectors attention scores FlashAttention SnapKV Streaming-LLM needle-in-a-haystack task generation perplexity