title: ChunkKV: Semantic-Preserving KV Cache Compression for Efficient
    Long-Context LLM Inference
summary: To reduce memory costs in long-context inference with Large Language Models
  (LLMs), many recent works focus on compressing the key-value (KV) cache of
  different tokens. However, we identify that the previous KV cache compression
  methods measure token importance individually, neglecting the dependency
  between different tokens in the real-world language characterics. In light of
  this, we introduce ChunkKV, grouping the tokens in a chunk as a basic
  compressing unit, and retaining the most informative semantic chunks while
  discarding the less important ones. Furthermore, observing that ChunkKV
  exhibits higher similarity in the preserved indices across different layers, we
  propose layer-wise index reuse to further reduce computational overhead. We
  evaluated ChunkKV on cutting-edge long-context benchmarks including LongBench
  and Needle-In-A-HayStack, as well as the GSM8K and JailbreakV in-context
  learning benchmark. Our experiments with instruction tuning and multi-step
  reasoning (O1 and R1) LLMs, achieve up to 10\% performance improvement under
  aggressive compression ratios compared to existing methods.
paper_authors._id: 67a1779d5f583199ce7921ad
paper_authors.name: Xiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T14:48:39.990Z
paper_authors._id: 63024676056ec3a2a8714b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
paper_authors.fullname: Xiang Liu
paper_authors.type: user
paper_authors.user: Dominic789654
paper_authors._id: 67a1779d5f583199ce7921ae
paper_authors.name: Zhenheng Tang
paper_authors._id: 67a1779d5f583199ce7921af
paper_authors.name: Peijie Dong
paper_authors._id: 67a1779d5f583199ce7921b0
paper_authors.name: Zeyu Li
paper_authors._id: 67a1779d5f583199ce7921b1
paper_authors.name: Bo Li
paper_authors._id: 67a1779d5f583199ce7921b2
paper_authors.name: Xuming Hu
paper_authors._id: 67a1779d5f583199ce7921b3
paper_authors.name: Xiaowen Chu
paper_title: ChunkKV: Semantic-Preserving KV Cache Compression for Efficient
    Long-Context LLM Inference
paper_summary: To reduce memory costs in long-context inference with Large Language Models
  (LLMs), many recent works focus on compressing the key-value (KV) cache of
  different tokens. However, we identify that the previous KV cache compression
  methods measure token importance individually, neglecting the dependency
  between different tokens in the real-world language characterics. In light of
  this, we introduce ChunkKV, grouping the tokens in a chunk as a basic
  compressing unit, and retaining the most informative semantic chunks while
  discarding the less important ones. Furthermore, observing that ChunkKV
  exhibits higher similarity in the preserved indices across different layers, we
  propose layer-wise index reuse to further reduce computational overhead. We
  evaluated ChunkKV on cutting-edge long-context benchmarks including LongBench
  and Needle-In-A-HayStack, as well as the GSM8K and JailbreakV in-context
  learning benchmark. Our experiments with instruction tuning and multi-step
  reasoning (O1 and R1) LLMs, achieve up to 10\% performance improvement under
  aggressive compression ratios compared to existing methods.
paper_ai_keywords: key-value (KV) cache token importance semantic chunks layer-wise index reuse LongBench Needle-In-A-HayStack GSM8K JailbreakV in-context learning instruction tuning multi-step reasoning