Corresponding author(s): ccharlie@deepmind.com
©2023 DeepMind. All rights reservedarXiv:2302.01318v1  [cs.CL]  2 Feb 2023
Accelerating Large Language Model Decoding with Speculative Sampling
draftingprocessforlargelanguagemodels,resultinginaneﬀectiveandpracticalmethodforreducing
sampling latency without the need for modifying the target model or biasing the sample distribution.
Depending on the evaluation domain, SpS leads to a 2–25speedup when sampling from Chinchilla
(Hoﬀmann et al., 2022). Notably, the mean tokens per second with SpS often exceeds the idealised
ceiling on auto-regressive sampling speed imposed by the memory bandwidth.
Related Work
There has been a substantial body of work focused on improving sampling latency of large transformers and other auto-regressive models.
Since sampling performance is heavily coupled with the model size in memory, quantisation to
int8or even int4(Dettmers et al., 2022; Yao et al., 2022) and distillation (Jiao et al., 2020; Sanh
et al., 2019) of transformers are eﬀective techniques for reducing sampling latency with little to no
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper