title: xKV: Cross-Layer SVD for KV-Cache Compression
summary: Large Language Models (LLMs) with long context windows enable powerful
  applications but come at the cost of high memory consumption to store the Key
  and Value states (KV-Cache). Recent studies attempted to merge KV-cache from
  multiple layers into shared representations, yet these approaches either
  require expensive pretraining or rely on assumptions of high per-token cosine
  similarity across layers which generally does not hold in practice. We find
  that the dominant singular vectors are remarkably well-aligned across multiple
  layers of the KV-Cache. Exploiting this insight, we propose xKV, a simple
  post-training method that applies Singular Value Decomposition (SVD) on the
  KV-Cache of grouped layers. xKV consolidates the KV-Cache of multiple layers
  into a shared low-rank subspace, significantly reducing KV-Cache sizes. Through
  extensive evaluations on the RULER long-context benchmark with widely-used LLMs
  (e.g., Llama-3.1 and Qwen2.5), xKV achieves up to 6.8x higher compression rates
  than state-of-the-art inter-layer technique while improving accuracy by 2.7%.
  Moreover, xKV is compatible with the emerging Multi-Head Latent Attention (MLA)
  (e.g., DeepSeek-Coder-V2), yielding a notable 3x compression rates on coding
  tasks without performance degradation. These results highlight xKV's strong
  capability and versatility in addressing memory bottlenecks for long-context
  LLM inference. Our code is publicly available at:
  https://github.com/abdelfattah-lab/xKV.
paper_authors._id: 67e4110756b46b70c6d49e2f
paper_authors.name: Chi-Chih Chang
paper_authors._id: 67e4110756b46b70c6d49e30
paper_authors.name: Chien-Yu Lin
paper_authors._id: 67e4110756b46b70c6d49e31
paper_authors.name: Yash Akhauri
paper_authors._id: 67e4110756b46b70c6d49e32
paper_authors.name: Wei-Cheng Lin
paper_authors._id: 67e4110756b46b70c6d49e33
paper_authors.name: Kai-Chiang Wu
paper_authors._id: 67e4110756b46b70c6d49e34
paper_authors.name: Luis Ceze
paper_authors._id: 67e4110756b46b70c6d49e35
paper_authors.name: Mohamed S. Abdelfattah
paper_title: xKV: Cross-Layer SVD for KV-Cache Compression
paper_summary: Large Language Models (LLMs) with long context windows enable powerful
  applications but come at the cost of high memory consumption to store the Key
  and Value states (KV-Cache). Recent studies attempted to merge KV-cache from
  multiple layers into shared representations, yet these approaches either
  require expensive pretraining or rely on assumptions of high per-token cosine
  similarity across layers which generally does not hold in practice. We find
  that the dominant singular vectors are remarkably well-aligned across multiple
  layers of the KV-Cache. Exploiting this insight, we propose xKV, a simple
  post-training method that applies Singular Value Decomposition (SVD) on the
  KV-Cache of grouped layers. xKV consolidates the KV-Cache of multiple layers
  into a shared low-rank subspace, significantly reducing KV-Cache sizes. Through
  extensive evaluations on the RULER long-context benchmark with widely-used LLMs
  (e.g., Llama-3.1 and Qwen2.5), xKV achieves up to 6.8x higher compression rates
  than state-of-the-art inter-layer technique while improving accuracy by 2.7%.
  Moreover, xKV is compatible with the emerging Multi-Head Latent Attention (MLA)
  (e.g., DeepSeek-Coder-V2), yielding a notable 3x compression rates on coding
  tasks without performance degradation. These results highlight xKV's strong
  capability and versatility in addressing memory bottlenecks for long-context
  LLM inference. Our code is publicly available at:
  https://github.com/abdelfattah-lab/xKV.
paper_ai_summary: xKV is a post-training method that uses Singular Value Decomposition to reduce the size of KV-Cache in LLMs with long context windows, improving compression rates and accuracy without significant performance degradation.
paper_ai_keywords: LLMs long context windows KV-Cache Singular Value Decomposition SVD RULER benchmark Multi-Head Latent Attention MLA DeepSeek-Coder-V2