KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
Transformer-based large language models (LLMs) cache context as key-value
(KV) pairs during inference. As context length grows, KV cache sizes expand,
leading to substantial memory overhead and increased attention latency. This
paper introduces KVzip, a query-agnostic KV cache eviction method enabling
effective reuse of compressed KV caches across diverse queries. KVzip
quantifies the importance of a KV pair using the underlying LLM to reconstruct
original contexts from cached KV pairs, subsequently evicting pairs with lower
importance. Extensive empirical evaluations demonstrate that KVzip reduces KV
cache size by 3-4times and FlashAttention decoding latency by approximately
2times, with negligible performance loss in question-answering, retrieval,
reasoning, and code comprehension tasks. Evaluations include various models
such as LLaMA3.1-8B, Qwen2.5-14B, and Gemma3-12B, with context lengths reaching
up to 170K tokens. KVzip significantly outperforms existing query-aware KV
eviction methods, which suffer from performance degradation even at a 90% cache
budget ratio under multi-query scenarios.
68393d075711daf8cc919c04
Jang-Hyun Kim
claimed_verified
2025-05-30T06:53:14.736Z
62e21907eda17fc126a15210
/avatars/097879cf1eafa5e77bde419c7172fac6.svg
Jang-Hyun Kim
user
Jang-Hyun
68393d075711daf8cc919c05
Jinuk Kim
claimed_verified
2025-05-30T06:53:17.350Z
60eb9074cc720726777d22a2
https://cdn-avatars.huggingface.co/v1/production/uploads/60eb9074cc720726777d22a2/aI-pHJRmnYOfC5fH7fFzD.jpeg
Jinuk Kim
user
jusjinuk
68393d075711daf8cc919c06
Sangwoo Kwon
68393d075711daf8cc919c07
Jae W. Lee
68393d075711daf8cc919c08
Sangdoo Yun
claimed_verified
2025-07-13T07:30:03.466Z
648ac415718bc0670a5a5f56
/avatars/27189e289b808ef01689ff2abb7a56bf.svg
Sangdoo Yun
user
oodgnas
68393d075711daf8cc919c09
Hyun Oh Song
KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
Transformer-based large language models (LLMs) cache context as key-value
(KV) pairs during inference. As context length grows, KV cache sizes expand,
leading to substantial memory overhead and increased attention latency. This
paper introduces KVzip, a query-agnostic KV cache eviction method enabling
effective reuse of compressed KV caches across diverse queries. KVzip
quantifies the importance of a KV pair using the underlying LLM to reconstruct
original contexts from cached KV pairs, subsequently evicting pairs with lower
importance. Extensive empirical evaluations demonstrate that KVzip reduces KV
cache size by 3-4times and FlashAttention decoding latency by approximately
2times, with negligible performance loss in question-answering, retrieval,
reasoning, and code comprehension tasks. Evaluations include various models
such as LLaMA3.1-8B, Qwen2.5-14B, and Gemma3-12B, with context lengths reaching
up to 170K tokens. KVzip significantly outperforms existing query-aware KV
eviction methods, which suffer from performance degradation even at a 90% cache
budget ratio under multi-query scenarios.
KVzip, a query-agnostic KV cache eviction method for transformer-based LLMs, reduces KV cache size and decoding latency while maintaining performance across various tasks and models.
Transformer-based large language models (LLMs) KV cache context length query-agnostic KVzip query-aware KV eviction attention latency FlashAttention question-answering retrieval reasoning code comprehension LLaMA3.1-8B Qwen2.5-14B Gemma3-12B