we call speculative sampling (SpS). This is achieved by:
1.Generating a short draft of length 𝐾. This can be attained with either a parallel model (Stern
et al., 2018) or by calling a faster, auto-regressive model 𝐾times. We shall refer to this model
as thedraft model , and focus on the case where it is auto-regressive.
2.Scoring the draft using the larger, more powerful model from we wish to sample from. We shall
refer to this model as the target model .
3.Using a modiﬁed rejection sampling scheme, accept a subset of the 𝐾draft tokens from left to
right, recovering the distribution of the target model in the process.
Intuitively, there are often sequences where the next token might be “obvious”. Therefore, if there is
strongagreementbetweenthedraftandtargetmodel’sdistributionsonagiventokenorsub-sequence
of tokens, this setup permits the generation of multiple tokens each time the target model is called .
We show that the expected acceptance rate of draft tokens is suﬃcient to oﬀset the overhead of the
Corresponding author(s): ccharlie@deepmind.com
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper