is the strategy used by Stern et al. (2018), which adds multiple heads into the transformer to
generate multiple tokens.
•Using sequence level distillation (Kim and Rush, 2016) to generate a second model which
predicts𝐾tokens in parallel. This strategy was employed by Ge et al. (2022).
•Set a portion of the activations of the target model as an input to the draft model, and train the
draft model with this input.
Although these methods will likely yield powerful drafts, they require a large number of data generated from the target model or changes to the target model. Sequence level distillation in particular
would require a large compute budget. This makes them less practical for large scale applications.
Whilst large language models produce better samples, intuitively there are "easier" tokens to predict
for which smaller models may be suﬃcient. Therefore we may simply use a smaller version of the
target language model as the draft and obtain high acceptance rates. This would also be convenient
from an engineering and workﬂow perspective, since robust tooling for such models should already
exist to train the target model in the ﬁrst place.
Results
We train a 4 billion parameter draft model optimised for sampling latency on 16 TPU v4s – the same
hardware that is typically used to serve Chinchilla for research purposes. This model was trained
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper