rejected, a valid token is resampled.
•Since the ﬁnal token of the draft gives us the logits for the next token, if every drafted token is
accepted, we can sample from it normally. This gives us a maximum of 𝐾¸1tokens per loop,
over the naive implementation which would only return 𝐾tokens.
With standard sampling methods such as nucleus, top-k sampling and adjusting temperature, we
can modify the probabilities accordingly before applying this rejection sampling scheme. We have
observed that the overall acceptance rate is robust to the exact parameters used.
Because we do not interact with the body of the transformer itself, this method can be used in
conjunction many other techniques for accelerating or optimising the memory use of sampling, such
as quantisation and multi-query attention.
Choice of Draft Models
Since the acceptance criterion guarantees the distribution of the target model in our samples, we are
free to choose the method for drafting a continuation as long as it exposes logits, and there is a high
enough acceptance rate and/or low enough latency to break-even. There exist several approaches
here:
•Incorporating draft generation into the target model, and train the model from the start. This
is the strategy used by Stern et al. (2018), which adds multiple heads into the transformer to
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper