title: KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
summary: Transformer-based large language models (LLMs) cache context as key-value
  (KV) pairs during inference. As context length grows, KV cache sizes expand,
  leading to substantial memory overhead and increased attention latency. This
  paper introduces KVzip, a query-agnostic KV cache eviction method enabling
  effective reuse of compressed KV caches across diverse queries. KVzip
  quantifies the importance of a KV pair using the underlying LLM to reconstruct
  original contexts from cached KV pairs, subsequently evicting pairs with lower
  importance. Extensive empirical evaluations demonstrate that KVzip reduces KV
  cache size by 3-4times and FlashAttention decoding latency by approximately
  2times, with negligible performance loss in question-answering, retrieval,
  reasoning, and code comprehension tasks. Evaluations include various models
  such as LLaMA3.1-8B, Qwen2.5-14B, and Gemma3-12B, with context lengths reaching
  up to 170K tokens. KVzip significantly outperforms existing query-aware KV
  eviction methods, which suffer from performance degradation even at a 90% cache
  budget ratio under multi-query scenarios.
paper_authors._id: 68393d075711daf8cc919c04
paper_authors.name: Jang-Hyun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:53:14.736Z
paper_authors._id: 62e21907eda17fc126a15210
paper_authors.avatarUrl: /avatars/097879cf1eafa5e77bde419c7172fac6.svg
paper_authors.fullname: Jang-Hyun Kim
paper_authors.type: user
paper_authors.user: Jang-Hyun
paper_authors._id: 68393d075711daf8cc919c05
paper_authors.name: Jinuk Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:53:17.350Z
paper_authors._id: 60eb9074cc720726777d22a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60eb9074cc720726777d22a2/aI-pHJRmnYOfC5fH7fFzD.jpeg
paper_authors.fullname: Jinuk Kim
paper_authors.type: user
paper_authors.user: jusjinuk
paper_authors._id: 68393d075711daf8cc919c06
paper_authors.name: Sangwoo Kwon
paper_authors._id: 68393d075711daf8cc919c07
paper_authors.name: Jae W. Lee
paper_authors._id: 68393d075711daf8cc919c08
paper_authors.name: Sangdoo Yun
paper_authors._id: 68393d075711daf8cc919c09
paper_authors.name: Hyun Oh Song
paper_title: KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
paper_summary: Transformer-based large language models (LLMs) cache context as key-value
  (KV) pairs during inference. As context length grows, KV cache sizes expand,
  leading to substantial memory overhead and increased attention latency. This
  paper introduces KVzip, a query-agnostic KV cache eviction method enabling
  effective reuse of compressed KV caches across diverse queries. KVzip
  quantifies the importance of a KV pair using the underlying LLM to reconstruct
  original contexts from cached KV pairs, subsequently evicting pairs with lower
  importance. Extensive empirical evaluations demonstrate that KVzip reduces KV
  cache size by 3-4times and FlashAttention decoding latency by approximately
  2times, with negligible performance loss in question-answering, retrieval,
  reasoning, and code comprehension tasks. Evaluations include various models
  such as LLaMA3.1-8B, Qwen2.5-14B, and Gemma3-12B, with context lengths reaching
  up to 170K tokens. KVzip significantly outperforms existing query-aware KV
  eviction methods, which suffer from performance degradation even at a 90% cache
  budget ratio under multi-query scenarios.
paper_ai_summary: KVzip, a query-agnostic KV cache eviction method for transformer-based LLMs, reduces KV cache size and decoding latency while maintaining performance across various tasks and models.
paper_ai_keywords: Transformer-based large language models (LLMs) KV cache context length query-agnostic KVzip query-aware KV eviction attention latency FlashAttention question-answering retrieval reasoning code comprehension LLaMA3.1-8B Qwen2.5-14B Gemma3-12B