title: Accelerating LLM Inference with Staged Speculative Decoding
summary: Recent advances with large language models (LLM) illustrate their diverse
  capabilities. We propose a novel algorithm, staged speculative decoding, to
  accelerate LLM inference in small-batch, on-device scenarios. We address the
  low arithmetic intensity of small-batch inference by improving upon previous
  work in speculative decoding. First, we restructure the speculative batch as a
  tree, which reduces generation costs and increases the expected tokens per
  batch. Second, we add a second stage of speculative decoding. Taken together,
  we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L
  model while perfectly preserving output quality.
paper_authors._id: 64d4498e0f3095b62d3b6440
paper_authors.name: Benjamin Spector
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-11T13:30:31.224Z
paper_authors._id: 63d074db623a3d1d117372ad
paper_authors.avatarUrl: /avatars/dbccd7cf1b2592bbc76dabcacf0a1e2f.svg
paper_authors.fullname: Benjamin Spector
paper_authors.type: user
paper_authors.user: benjaminfspector
paper_authors._id: 64d4498e0f3095b62d3b6441
paper_authors.name: Chris Re
paper_title: Accelerating LLM Inference with Staged Speculative Decoding
paper_summary: Recent advances with large language models (LLM) illustrate their diverse
  capabilities. We propose a novel algorithm, staged speculative decoding, to
  accelerate LLM inference in small-batch, on-device scenarios. We address the
  low arithmetic intensity of small-batch inference by improving upon previous
  work in speculative decoding. First, we restructure the speculative batch as a
  tree, which reduces generation costs and increases the expected tokens per
  batch. Second, we add a second stage of speculative decoding. Taken together,
  we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L
  model while perfectly preserving output quality.
paper_ai_keywords: speculative decoding low arithmetic intensity speculative batch tree structure