title: Speculative Streaming: Fast LLM Inference without Auxiliary Models
summary: Speculative decoding is a prominent technique to speed up the inference of a
  large target language model based on predictions of an auxiliary draft model.
  While effective, in application-specific settings, it often involves
  fine-tuning both draft and target models to achieve high acceptance rates. As
  the number of downstream tasks grows, these draft models add significant
  complexity to inference systems. We propose Speculative Streaming, a
  single-model speculative decoding method that fuses drafting into the target
  model by changing the fine-tuning objective from next token prediction to
  future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 -
  3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and
  Meaning Representation, without sacrificing generation quality. Additionally,
  Speculative Streaming is parameter-efficient. It achieves on-par/higher
  speed-ups than Medusa-style architectures while using ~10000X fewer extra
  parameters, making it well-suited for resource-constrained devices.
paper_authors._id: 65d438710cf70a406c627722
paper_authors.name: Nikhil Bhendawade
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:30:34.595Z
paper_authors._id: 64d5061ea146b1c0a63f91c6
paper_authors.avatarUrl: /avatars/41f322d00c859cd345b853b55a3ad59f.svg
paper_authors.fullname: Nikhil
paper_authors.type: user
paper_authors.user: NickNickGo
paper_authors._id: 65d438710cf70a406c627723
paper_authors.name: Irina Belousova
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T18:24:19.029Z
paper_authors._id: 65d4e7cbc81e8c3cffafa8af
paper_authors.avatarUrl: /avatars/aa490b7e103320c8951b1c3beb242e1d.svg
paper_authors.fullname: Irina Belousova
paper_authors.type: user
paper_authors.user: ibelousova
paper_authors._id: 65d438710cf70a406c627724
paper_authors.name: Qichen Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:45:37.162Z
paper_authors._id: 6496114d33611fb433f16970
paper_authors.avatarUrl: /avatars/ac722222c39e87c772705a3bf52118ad.svg
paper_authors.fullname: Qichen Fu
paper_authors.type: user
paper_authors.user: janemiminini
paper_authors._id: 65d438710cf70a406c627725
paper_authors.name: Henry Mason
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T18:24:21.752Z
paper_authors._id: 630d189254c3dbd48056bf63
paper_authors.avatarUrl: /avatars/6b7a3dd235be1ba5679775ca53c4116d.svg
paper_authors.fullname: Henry Mason
paper_authors.type: user
paper_authors.user: docgok
paper_authors._id: 65d438710cf70a406c627726
paper_authors.name: Mohammad Rastegari
paper_authors._id: 65d438710cf70a406c627727
paper_authors.name: Mahyar Najibi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-15T21:44:36.448Z
paper_authors._id: 661d992d69a65c0fb0717013
paper_authors.avatarUrl: /avatars/915d593a87111bca85e3500500246345.svg
paper_authors.fullname: Mahyar Najibi
paper_authors.type: user
paper_authors.user: mahyar-najibi
paper_title: Speculative Streaming: Fast LLM Inference without Auxiliary Models
paper_summary: Speculative decoding is a prominent technique to speed up the inference of a
  large target language model based on predictions of an auxiliary draft model.
  While effective, in application-specific settings, it often involves
  fine-tuning both draft and target models to achieve high acceptance rates. As
  the number of downstream tasks grows, these draft models add significant
  complexity to inference systems. We propose Speculative Streaming, a
  single-model speculative decoding method that fuses drafting into the target
  model by changing the fine-tuning objective from next token prediction to
  future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 -
  3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and
  Meaning Representation, without sacrificing generation quality. Additionally,
  Speculative Streaming is parameter-efficient. It achieves on-par/higher
  speed-ups than Medusa-style architectures while using ~10000X fewer extra
  parameters, making it well-suited for resource-constrained devices.
paper_ai_keywords: speculative decoding inference large target language model auxiliary draft model fine-tuning n-gram prediction Speculative Streaming Summarization structured Queries Meaning Representation parameter-efficient Medusa-style architectures