title: ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM
    Inference
summary: With the widespread deployment of long-context large language models (LLMs),
  there has been a growing demand for efficient support of high-throughput
  inference. However, as the key-value (KV) cache expands with the sequence
  length, the increasing memory footprint and the need to access it for each
  token generation both result in low throughput when serving long-context LLMs.
  While various dynamic sparse attention methods have been proposed to speed up
  inference while maintaining generation quality, they either fail to
  sufficiently reduce GPU memory consumption or introduce significant decoding
  latency by offloading the KV cache to the CPU. We present ShadowKV, a
  high-throughput long-context LLM inference system that stores the low-rank key
  cache and offloads the value cache to reduce the memory footprint for larger
  batch sizes and longer sequences. To minimize decoding latency, ShadowKV
  employs an accurate KV selection strategy that reconstructs minimal sparse KV
  pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks,
  including RULER, LongBench, and Needle In A Haystack, and models like
  Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and
  Qwen2-7B-128K, we demonstrate that it can support up to 6times larger batch
  sizes and boost throughput by up to 3.04times on an A100 GPU without
  sacrificing accuracy, even surpassing the performance achievable with infinite
  batch size under the assumption of infinite GPU memory. The code is available
  at https://github.com/bytedance/ShadowKV.
paper_authors._id: 67219228541e69bc0e6d3b6b
paper_authors.name: Hanshi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:51.173Z
paper_authors._id: 651b23a0a6e00a1678c9d8af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651b23a0a6e00a1678c9d8af/_Ip-ysvcjGpFCI9HtcTgx.jpeg
paper_authors.fullname: Hanshi
paper_authors.type: user
paper_authors.user: preminstrel
paper_authors._id: 67219228541e69bc0e6d3b6c
paper_authors.name: Li-Wen Chang
paper_authors._id: 67219228541e69bc0e6d3b6d
paper_authors.name: Wenlei Bao
paper_authors._id: 67219228541e69bc0e6d3b6e
paper_authors.name: Size Zheng
paper_authors._id: 67219228541e69bc0e6d3b6f
paper_authors.name: Ningxin Zheng
paper_authors._id: 67219228541e69bc0e6d3b70
paper_authors.name: Xin Liu
paper_authors._id: 67219228541e69bc0e6d3b71
paper_authors.name: Harry Dong
paper_authors._id: 67219228541e69bc0e6d3b72
paper_authors.name: Yuejie Chi
paper_authors._id: 67219228541e69bc0e6d3b73
paper_authors.name: Beidi Chen
paper_title: ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM
    Inference
paper_summary: With the widespread deployment of long-context large language models (LLMs),
  there has been a growing demand for efficient support of high-throughput
  inference. However, as the key-value (KV) cache expands with the sequence
  length, the increasing memory footprint and the need to access it for each
  token generation both result in low throughput when serving long-context LLMs.
  While various dynamic sparse attention methods have been proposed to speed up
  inference while maintaining generation quality, they either fail to
  sufficiently reduce GPU memory consumption or introduce significant decoding
  latency by offloading the KV cache to the CPU. We present ShadowKV, a
  high-throughput long-context LLM inference system that stores the low-rank key
  cache and offloads the value cache to reduce the memory footprint for larger
  batch sizes and longer sequences. To minimize decoding latency, ShadowKV
  employs an accurate KV selection strategy that reconstructs minimal sparse KV
  pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks,
  including RULER, LongBench, and Needle In A Haystack, and models like
  Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and
  Qwen2-7B-128K, we demonstrate that it can support up to 6times larger batch
  sizes and boost throughput by up to 3.04times on an A100 GPU without
  sacrificing accuracy, even surpassing the performance achievable with infinite
  batch size under the assumption of infinite GPU memory. The code is available
  at https://github.com/bytedance/ShadowKV.
paper_ai_keywords: key-value (KV) cache dynamic sparse attention GPU memory consumption decoding latency low-rank key cache value cache minimal sparse KV pairs RULER LongBench Needle In A Haystack Llama-3.1-8B Llama-3-8B-1M GLM-4-9B-1M Yi-9B-200K Phi-3-Mini-128K Qwen2-7B-128K infinite batch size infinite GPU memory