et al., 2019) of transformers are eﬀective techniques for reducing sampling latency with little to no
performance penalty. The observation that model size contributes less to the ﬁnal performance than
expected (Hoﬀmann et al., 2022) has also encouraged smaller language models in general.
During sampling, a cache of the keys and values is maintained for every attention layer, and could
become a memory bandwidth bottleneck as the batch size increases. Methods such as multi-query
attention (Shazeer, 2019) aims to improve sampling performance by shrinking this cache. However
thesetechniquesaremosteﬀectiveatmaximisingthroughout(atlargerbatchsizes)insteadoflatency,
especially for larger models where the majority of the memory bandwidth budget is consumed by the
parameters.
Using a combination of the above techniques, in addition to a number of low-level optimisations to
TPUs, Pope et al. (2022) have greatly improved the serving latency and eﬃciency of PaLM 540B.
There is an existing body of similar work exploiting the eﬃciency of transformers and sequence
models operating in parallel. This includes block parallel sampling (Stern et al., 2018), aggressive
decoding (Ge et al., 2022), in addition to some work in parallelizing autoregressive models in the
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper