end while
Speculative Sampling
Conditional Scoring
For speculative sampling (See algorithm 2), we ﬁrst make the observation that computing the logits
of a short continuation of 𝐾tokens in parallel has a very similar latency to that of sampling a single
3
Accelerating Large Language Model Decoding with Speculative Sampling
token. We focus our attention on large transformers, sharded in the Megatron style (Shoeybi et al.,
2019). For these models the majority of sampling time can be attributed to three components:
1.Linear Layers: For small batch sizes, each linear layer only processes a small number of
embeddings. This causes the dense matrix multiplies in the feed-forward layers, queries, keys,
values computations and the ﬁnal attention projection to become memory bound. For small 𝐾,
this will continue to be memory bound and therefore take a similar amount of time.
2.The Attention Mechanism: Theattentionmechanismisalsomemorybound. Duringsampling,
we maintain a cache of all the keys and values of the previous tokens in the sequence to avoid
re-computation. These KV-caches are large, and accounts for the majority of the memory
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper