title: SCBench: A KV Cache-Centric Analysis of Long-Context Methods
summary: Long-context LLMs have enabled numerous downstream applications but also
  introduced significant challenges related to computational and memory
  efficiency. To address these challenges, optimizations for long-context
  inference have been developed, centered around the KV cache. However, existing
  benchmarks often evaluate in single-request, neglecting the full lifecycle of
  the KV cache in real-world use. This oversight is particularly critical, as KV
  cache reuse has become widely adopted in LLMs inference frameworks, such as
  vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft,
  Google, and Anthropic. To address this gap, we introduce
  SCBench(SharedContextBench), a comprehensive benchmark for evaluating
  long-context methods from a KV cachecentric perspective: 1) KV cache
  generation, 2) KV cache compression, 3) KV cache retrieval, 4) KV cache
  loading. Specifically, SCBench uses test examples with shared context, ranging
  12 tasks with two shared context modes, covering four categories of
  long-context capabilities: string retrieval, semantic retrieval, global
  information, and multi-task. With it, we provide an extensive KV cache-centric
  analysis of eight categories long-context solutions, including Gated Linear
  RNNs, Mamba-Attention hybrids, and efficient methods such as sparse attention,
  KV cache dropping, quantization, retrieval, loading, and prompt compression.
  The evaluation is conducted on 8 long-context LLMs. Our findings show that
  sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding
  with O(n) memory and sub-O(n^2) pre-filling computation perform robustly.
  Dynamic sparsity yields more expressive KV caches than static patterns, and
  layer-level sparsity in hybrid architectures reduces memory usage with strong
  performance. Additionally, we identify attention distribution shift issues in
  long-generation scenarios. https://aka.ms/SCBench.
paper_authors._id: 675f918be24babdf1816c11e
paper_authors.name: Yucheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:23:35.424Z
paper_authors._id: 63d00710645dd8d34ea9bcc6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d00710645dd8d34ea9bcc6/E6YsIsXH57OACL-NZ52fB.jpeg
paper_authors.fullname: Yucheng
paper_authors.type: user
paper_authors.user: liyucheng
paper_authors._id: 675f918be24babdf1816c11f
paper_authors.name: Huiqiang Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:22:52.093Z
paper_authors._id: 6278bd42541f3d2dfa77ea70
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
paper_authors.fullname: Huiqiang Jiang
paper_authors.type: user
paper_authors.user: iofu728
paper_authors._id: 675f918be24babdf1816c120
paper_authors.name: Qianhui Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:22:43.253Z
paper_authors._id: 63ef330b1e695b35aa484e11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
paper_authors.fullname: Qianhui WU
paper_authors.type: user
paper_authors.user: qianhuiwu
paper_authors._id: 675f918be24babdf1816c121
paper_authors.name: Xufang Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:22:37.186Z
paper_authors._id: 64b750a2fdb702b3d8619514
paper_authors.avatarUrl: /avatars/f09181c0825763dff692c4bc65effc4c.svg
paper_authors.fullname: Xufang Luo
paper_authors.type: user
paper_authors.user: luoxufang
paper_authors._id: 675f918be24babdf1816c122
paper_authors.name: Surin Ahn
paper_authors._id: 675f918be24babdf1816c123
paper_authors.name: Chengruidong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:22:07.437Z
paper_authors._id: 64646896884f2e3e1ced3cd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chengruidong
paper_authors._id: 675f918be24babdf1816c124
paper_authors.name: Amir H. Abdi
paper_authors._id: 675f918be24babdf1816c125
paper_authors.name: Dongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:21:34.011Z
paper_authors._id: 6524f13821af4b07309015aa
paper_authors.avatarUrl: /avatars/f97d6a041428d48305568df7afd83cc8.svg
paper_authors.fullname: dongsheng li
paper_authors.type: user
paper_authors.user: dongshengli
paper_authors._id: 675f918be24babdf1816c126
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:21:06.943Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 675f918be24babdf1816c127
paper_authors.name: Yuqing Yang
paper_authors._id: 675f918be24babdf1816c128
paper_authors.name: Lili Qiu
paper_title: SCBench: A KV Cache-Centric Analysis of Long-Context Methods
paper_summary: Long-context LLMs have enabled numerous downstream applications but also
  introduced significant challenges related to computational and memory
  efficiency. To address these challenges, optimizations for long-context
  inference have been developed, centered around the KV cache. However, existing
  benchmarks often evaluate in single-request, neglecting the full lifecycle of
  the KV cache in real-world use. This oversight is particularly critical, as KV
  cache reuse has become widely adopted in LLMs inference frameworks, such as
  vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft,
  Google, and Anthropic. To address this gap, we introduce
  SCBench(SharedContextBench), a comprehensive benchmark for evaluating
  long-context methods from a KV cachecentric perspective: 1) KV cache
  generation, 2) KV cache compression, 3) KV cache retrieval, 4) KV cache
  loading. Specifically, SCBench uses test examples with shared context, ranging
  12 tasks with two shared context modes, covering four categories of
  long-context capabilities: string retrieval, semantic retrieval, global
  information, and multi-task. With it, we provide an extensive KV cache-centric
  analysis of eight categories long-context solutions, including Gated Linear
  RNNs, Mamba-Attention hybrids, and efficient methods such as sparse attention,
  KV cache dropping, quantization, retrieval, loading, and prompt compression.
  The evaluation is conducted on 8 long-context LLMs. Our findings show that
  sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding
  with O(n) memory and sub-O(n^2) pre-filling computation perform robustly.
  Dynamic sparsity yields more expressive KV caches than static patterns, and
  layer-level sparsity in hybrid architectures reduces memory usage with strong
  performance. Additionally, we identify attention distribution shift issues in
  long-generation scenarios. https://aka.ms/SCBench.
paper_ai_keywords: KV cache vLLM SGLang Gated Linear RNNs Mamba-Attention hybrids sparse attention KV cache dropping quantization retrieval loading prompt compression long-context LLMs string retrieval semantic retrieval global information multi-task sub-O(n) memory methods O(n) memory sub-O(n^2) pre-filling computation dynamic sparsity static patterns layer-level sparsity