title: MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context
    Generation with Speculative Decoding
summary: Large Language Models (LLMs) have become more prevalent in long-context
  applications such as interactive chatbots, document analysis, and agent
  workflows, but it is challenging to serve long-context requests with low
  latency and high throughput. Speculative decoding (SD) is a widely used
  technique to reduce latency without sacrificing performance but the
  conventional wisdom suggests that its efficacy is limited to small batch sizes.
  In MagicDec, we show that surprisingly SD can achieve speedup even for a high
  throughput inference regime for moderate to long sequences. More interestingly,
  an intelligent drafting strategy can achieve better speedup with increasing
  batch size based on our rigorous analysis. MagicDec first identifies the
  bottleneck shifts with increasing batch size and sequence length, and uses
  these insights to deploy speculative decoding more effectively for high
  throughput inference. Then, it leverages draft models with sparse KV cache to
  address the KV bottleneck that scales with both sequence length and batch size.
paper_authors._id: 66c557a6e9d46e06d5d498d6
paper_authors.name: Jian Chen
paper_authors._id: 66c557a6e9d46e06d5d498d7
paper_authors.name: Vashisth Tiwari
paper_authors._id: 66c557a6e9d46e06d5d498d8
paper_authors.name: Ranajoy Sadhukhan
paper_authors._id: 66c557a6e9d46e06d5d498d9
paper_authors.name: Zhuoming Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:43:43.763Z
paper_authors._id: 650ddecfdf31b1809f12a273
paper_authors.avatarUrl: /avatars/2c6300471cd09a067d56aa09bc06ad78.svg
paper_authors.fullname: ZMC
paper_authors.type: user
paper_authors.user: ZMC2019
paper_authors._id: 66c557a6e9d46e06d5d498da
paper_authors.name: Jinyuan Shi
paper_authors._id: 66c557a6e9d46e06d5d498db
paper_authors.name: Ian En-Hsu Yen
paper_authors._id: 66c557a6e9d46e06d5d498dc
paper_authors.name: Beidi Chen
paper_title: MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context
    Generation with Speculative Decoding
paper_summary: Large Language Models (LLMs) have become more prevalent in long-context
  applications such as interactive chatbots, document analysis, and agent
  workflows, but it is challenging to serve long-context requests with low
  latency and high throughput. Speculative decoding (SD) is a widely used
  technique to reduce latency without sacrificing performance but the
  conventional wisdom suggests that its efficacy is limited to small batch sizes.
  In MagicDec, we show that surprisingly SD can achieve speedup even for a high
  throughput inference regime for moderate to long sequences. More interestingly,
  an intelligent drafting strategy can achieve better speedup with increasing
  batch size based on our rigorous analysis. MagicDec first identifies the
  bottleneck shifts with increasing batch size and sequence length, and uses
  these insights to deploy speculative decoding more effectively for high
  throughput inference. Then, it leverages draft models with sparse KV cache to
  address the KV bottleneck that scales with both sequence length and batch size.
paper_ai_summary: MagicDec improves high-throughput inference for long-context LLMs by optimizing speculative decoding and using draft models with sparse KV cache.
paper_ai_keywords: LLMs long-context applications speculative decoding high-throughput inference batch sizes sequence length bottleneck KV cache draft models