MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context
  Generation with Speculative Decoding
Large Language Models (LLMs) have become more prevalent in long-context
applications such as interactive chatbots, document analysis, and agent
workflows, but it is challenging to serve long-context requests with low
latency and high throughput. Speculative decoding (SD) is a widely used
technique to reduce latency without sacrificing performance but the
conventional wisdom suggests that its efficacy is limited to small batch sizes.
In MagicDec, we show that surprisingly SD can achieve speedup even for a high
throughput inference regime for moderate to long sequences. More interestingly,
an intelligent drafting strategy can achieve better speedup with increasing
batch size based on our rigorous analysis. MagicDec first identifies the
bottleneck shifts with increasing batch size and sequence length, and uses
these insights to deploy speculative decoding more effectively for high
throughput inference. Then, it leverages draft models with sparse KV cache to
address the KV bottleneck that scales with both sequence length and batch size.
66c557a6e9d46e06d5d498d6
Jian Chen
66c557a6e9d46e06d5d498d7
Vashisth Tiwari
66c557a6e9d46e06d5d498d8
Ranajoy Sadhukhan
66c557a6e9d46e06d5d498d9
Zhuoming Chen
claimed_verified
2025-06-06T07:43:43.763Z
650ddecfdf31b1809f12a273
/avatars/2c6300471cd09a067d56aa09bc06ad78.svg
ZMC
user
ZMC2019
66c557a6e9d46e06d5d498da
Jinyuan Shi
66c557a6e9d46e06d5d498db
Ian En-Hsu Yen
66c557a6e9d46e06d5d498dc
Beidi Chen
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context
  Generation with Speculative Decoding
Large Language Models (LLMs) have become more prevalent in long-context
applications such as interactive chatbots, document analysis, and agent
workflows, but it is challenging to serve long-context requests with low
latency and high throughput. Speculative decoding (SD) is a widely used
technique to reduce latency without sacrificing performance but the
conventional wisdom suggests that its efficacy is limited to small batch sizes.
In MagicDec, we show that surprisingly SD can achieve speedup even for a high
throughput inference regime for moderate to long sequences. More interestingly,
an intelligent drafting strategy can achieve better speedup with increasing
batch size based on our rigorous analysis. MagicDec first identifies the
bottleneck shifts with increasing batch size and sequence length, and uses
these insights to deploy speculative decoding more effectively for high
throughput inference. Then, it leverages draft models with sparse KV cache to
address the KV bottleneck that scales with both sequence length and batch size.
MagicDec improves high-throughput inference for long-context LLMs by optimizing speculative decoding and using draft models with sparse KV cache.
LLMs long-context applications speculative decoding high-throughput inference batch sizes sequence length bottleneck KV cache draft models