ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM
  Inference
With the widespread deployment of long-context large language models (LLMs),
there has been a growing demand for efficient support of high-throughput
inference. However, as the key-value (KV) cache expands with the sequence
length, the increasing memory footprint and the need to access it for each
token generation both result in low throughput when serving long-context LLMs.
While various dynamic sparse attention methods have been proposed to speed up
inference while maintaining generation quality, they either fail to
sufficiently reduce GPU memory consumption or introduce significant decoding
latency by offloading the KV cache to the CPU. We present ShadowKV, a
high-throughput long-context LLM inference system that stores the low-rank key
cache and offloads the value cache to reduce the memory footprint for larger
batch sizes and longer sequences. To minimize decoding latency, ShadowKV
employs an accurate KV selection strategy that reconstructs minimal sparse KV
pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks,
including RULER, LongBench, and Needle In A Haystack, and models like
Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and
Qwen2-7B-128K, we demonstrate that it can support up to 6times larger batch
sizes and boost throughput by up to 3.04times on an A100 GPU without
sacrificing accuracy, even surpassing the performance achievable with infinite
batch size under the assumption of infinite GPU memory. The code is available
at https://github.com/bytedance/ShadowKV.
67219228541e69bc0e6d3b6b
Hanshi Sun
claimed_verified
2024-10-30T09:52:51.173Z
651b23a0a6e00a1678c9d8af
https://cdn-avatars.huggingface.co/v1/production/uploads/651b23a0a6e00a1678c9d8af/_Ip-ysvcjGpFCI9HtcTgx.jpeg
Hanshi
user
preminstrel
67219228541e69bc0e6d3b6c
Li-Wen Chang
67219228541e69bc0e6d3b6d
Wenlei Bao
67219228541e69bc0e6d3b6e
Size Zheng
67219228541e69bc0e6d3b6f
Ningxin Zheng
67219228541e69bc0e6d3b70
Xin Liu
67219228541e69bc0e6d3b71
Harry Dong
67219228541e69bc0e6d3b72
Yuejie Chi
67219228541e69bc0e6d3b73
Beidi Chen
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM
  Inference
With the widespread deployment of long-context large language models (LLMs),
there has been a growing demand for efficient support of high-throughput
inference. However, as the key-value (KV) cache expands with the sequence
length, the increasing memory footprint and the need to access it for each
token generation both result in low throughput when serving long-context LLMs.
While various dynamic sparse attention methods have been proposed to speed up
inference while maintaining generation quality, they either fail to
sufficiently reduce GPU memory consumption or introduce significant decoding
latency by offloading the KV cache to the CPU. We present ShadowKV, a
high-throughput long-context LLM inference system that stores the low-rank key
cache and offloads the value cache to reduce the memory footprint for larger
batch sizes and longer sequences. To minimize decoding latency, ShadowKV
employs an accurate KV selection strategy that reconstructs minimal sparse KV
pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks,
including RULER, LongBench, and Needle In A Haystack, and models like
Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and
Qwen2-7B-128K, we demonstrate that it can support up to 6times larger batch
sizes and boost throughput by up to 3.04times on an A100 GPU without
sacrificing accuracy, even surpassing the performance achievable with infinite
batch size under the assumption of infinite GPU memory. The code is available
at https://github.com/bytedance/ShadowKV.
ShadowKV is a high-throughput inference system for long-context LLMs that reduces memory footprint and minimizes decoding latency by using a low-rank key cache and minimal sparse KV pair reconstruction.
long-context LLMs key-value (KV) cache sparse attention GPU memory consumption decoding latency low-rank key cache sparse KV pairs RULER LongBench Needle In A Haystack Llama-3.1-8B Llama-3-8B-1M GLM-4-9B-1M Yi-9B-200K Phi-3-Mini-128K Qwen2-7B-128K