SCBench: A KV Cache-Centric Analysis of Long-Context Methods
Long-context LLMs have enabled numerous downstream applications but also
introduced significant challenges related to computational and memory
efficiency. To address these challenges, optimizations for long-context
inference have been developed, centered around the KV cache. However, existing
benchmarks often evaluate in single-request, neglecting the full lifecycle of
the KV cache in real-world use. This oversight is particularly critical, as KV
cache reuse has become widely adopted in LLMs inference frameworks, such as
vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft,
Google, and Anthropic. To address this gap, we introduce
SCBench(SharedContextBench), a comprehensive benchmark for evaluating
long-context methods from a KV cachecentric perspective: 1) KV cache
generation, 2) KV cache compression, 3) KV cache retrieval, 4) KV cache
loading. Specifically, SCBench uses test examples with shared context, ranging
12 tasks with two shared context modes, covering four categories of
long-context capabilities: string retrieval, semantic retrieval, global
information, and multi-task. With it, we provide an extensive KV cache-centric
analysis of eight categories long-context solutions, including Gated Linear
RNNs, Mamba-Attention hybrids, and efficient methods such as sparse attention,
KV cache dropping, quantization, retrieval, loading, and prompt compression.
The evaluation is conducted on 8 long-context LLMs. Our findings show that
sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding
with O(n) memory and sub-O(n^2) pre-filling computation perform robustly.
Dynamic sparsity yields more expressive KV caches than static patterns, and
layer-level sparsity in hybrid architectures reduces memory usage with strong
performance. Additionally, we identify attention distribution shift issues in
long-generation scenarios. https://aka.ms/SCBench.
675f918be24babdf1816c11e
Yucheng Li
admin_assigned
2024-12-16T10:23:35.424Z
63d00710645dd8d34ea9bcc6
https://cdn-avatars.huggingface.co/v1/production/uploads/63d00710645dd8d34ea9bcc6/E6YsIsXH57OACL-NZ52fB.jpeg
Yucheng
user
liyucheng
675f918be24babdf1816c11f
Huiqiang Jiang
admin_assigned
2024-12-16T10:22:52.093Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
675f918be24babdf1816c120
Qianhui Wu
admin_assigned
2024-12-16T10:22:43.253Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
user
qianhuiwu
675f918be24babdf1816c121
Xufang Luo
admin_assigned
2024-12-16T10:22:37.186Z
64b750a2fdb702b3d8619514
/avatars/f09181c0825763dff692c4bc65effc4c.svg
Xufang Luo
user
luoxufang
675f918be24babdf1816c122
Surin Ahn
675f918be24babdf1816c123
Chengruidong Zhang
admin_assigned
2024-12-16T10:22:07.437Z
64646896884f2e3e1ced3cd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
Zhang
user
Chengruidong
675f918be24babdf1816c124
Amir H. Abdi
675f918be24babdf1816c125
Dongsheng Li
admin_assigned
2024-12-16T10:21:34.011Z
6524f13821af4b07309015aa
/avatars/f97d6a041428d48305568df7afd83cc8.svg
dongsheng li
user
dongshengli
675f918be24babdf1816c126
Jianfeng Gao
admin_assigned
2024-12-16T10:21:06.943Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
675f918be24babdf1816c127
Yuqing Yang
675f918be24babdf1816c128
Lili Qiu
SCBench: A KV Cache-Centric Analysis of Long-Context Methods
Long-context LLMs have enabled numerous downstream applications but also
introduced significant challenges related to computational and memory
efficiency. To address these challenges, optimizations for long-context
inference have been developed, centered around the KV cache. However, existing
benchmarks often evaluate in single-request, neglecting the full lifecycle of
the KV cache in real-world use. This oversight is particularly critical, as KV
cache reuse has become widely adopted in LLMs inference frameworks, such as
vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft,
Google, and Anthropic. To address this gap, we introduce
SCBench(SharedContextBench), a comprehensive benchmark for evaluating
long-context methods from a KV cachecentric perspective: 1) KV cache
generation, 2) KV cache compression, 3) KV cache retrieval, 4) KV cache
loading. Specifically, SCBench uses test examples with shared context, ranging
12 tasks with two shared context modes, covering four categories of
long-context capabilities: string retrieval, semantic retrieval, global
information, and multi-task. With it, we provide an extensive KV cache-centric
analysis of eight categories long-context solutions, including Gated Linear
RNNs, Mamba-Attention hybrids, and efficient methods such as sparse attention,
KV cache dropping, quantization, retrieval, loading, and prompt compression.
The evaluation is conducted on 8 long-context LLMs. Our findings show that
sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding
with O(n) memory and sub-O(n^2) pre-filling computation perform robustly.
Dynamic sparsity yields more expressive KV caches than static patterns, and
layer-level sparsity in hybrid architectures reduces memory usage with strong
performance. Additionally, we identify attention distribution shift issues in
long-generation scenarios. https://aka.ms/SCBench.
SCBench evaluates KV cache-centric performance of long-context methods in LLMs, covering cache generation, compression, retrieval, and loading, highlighting robustness of sparse attention methods.
KV cache Gated Linear RNNs Mamba-Attention hybrids sparse attention KV cache dropping quantization retrieval loading prompt compression layer-level sparsity attention distribution shift