title: Inference-Time Hyper-Scaling with KV Cache Compression
summary: Inference-time scaling trades efficiency for increased reasoning accuracy by
  generating longer or more parallel sequences. However, in Transformer LLMs,
  generation cost is bottlenecked by the size of the key-value (KV) cache, rather
  than the number of generated tokens. Hence, we explore inference-time
  hyper-scaling: by compressing the KV cache, we can generate more tokens within
  the same compute budget and further improve the accuracy of scaled inference.
  The success of this approach, however, hinges on the ability of compression
  methods to preserve accuracy even at high compression ratios. To make
  hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), a
  novel method for sparsifying KV caches that only requires 1K training steps to
  achieve 8times compression, while maintaining better accuracy than
  training-free sparse attention. Instead of prematurely discarding cached
  tokens, DMS delays token eviction, implicitly merging representations and
  preserving critical information. We demonstrate the effectiveness of
  inference-time hyper-scaling with DMS on multiple families of LLMs, showing
  that it boosts accuracy for comparable inference runtime and memory load. For
  instance, we enhance Qwen-R1 32B by an average of 9.1 points on AIME 24, 7.6 on
  GPQA, and 9.6 on LiveCodeBench across compute budgets.
paper_authors._id: 6842a3cb9393cafb521855aa
paper_authors.name: Adrian Łańcucki
paper_authors._id: 6842a3cb9393cafb521855ab
paper_authors.name: Konrad Staniszewski
paper_authors._id: 6842a3cb9393cafb521855ac
paper_authors.name: Piotr Nawrot
paper_authors._id: 6842a3cb9393cafb521855ad
paper_authors.name: Edoardo M. Ponti
paper_title: Inference-Time Hyper-Scaling with KV Cache Compression
paper_summary: Inference-time scaling trades efficiency for increased reasoning accuracy by
  generating longer or more parallel sequences. However, in Transformer LLMs,
  generation cost is bottlenecked by the size of the key-value (KV) cache, rather
  than the number of generated tokens. Hence, we explore inference-time
  hyper-scaling: by compressing the KV cache, we can generate more tokens within
  the same compute budget and further improve the accuracy of scaled inference.
  The success of this approach, however, hinges on the ability of compression
  methods to preserve accuracy even at high compression ratios. To make
  hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), a
  novel method for sparsifying KV caches that only requires 1K training steps to
  achieve 8times compression, while maintaining better accuracy than
  training-free sparse attention. Instead of prematurely discarding cached
  tokens, DMS delays token eviction, implicitly merging representations and
  preserving critical information. We demonstrate the effectiveness of
  inference-time hyper-scaling with DMS on multiple families of LLMs, showing
  that it boosts accuracy for comparable inference runtime and memory load. For
  instance, we enhance Qwen-R1 32B by an average of 9.1 points on AIME 24, 7.6 on
  GPQA, and 9.6 on LiveCodeBench across compute budgets.
paper_ai_summary: Inference-time hyper-scaling with Dynamic Memory Sparsification in Transformer LLMs allows for increased token generation within the same compute budget by compressing the key-value cache, thereby enhancing accuracy.
paper_ai_keywords: inference-time hyper-scaling key-value (KV) cache Dynamic Memory Sparsification (DMS) token eviction representation merging AIME 24 GPQA LiveCodeBench Qwen-R1 32B