decoding (Ge et al., 2022), in addition to some work in parallelizing autoregressive models in the
image domain (Song et al., 2021; Wiggers and Hoogeboom, 2020). These methods have yet to be
adapted to typical language model use-cases since they either only work with greedy sampling, bias
the results or are focused on other modalities. Further, to our knowledge none of these techniques
have been scaled to distributed setups, which is necessary for the most expensive decoders with the
tens or hundreds of billions of parameters.
Coincidentally, the work in this manuscript was undertaken concurrently and independently of
the work on speculative decoding from Leviathan et al. (2022). We focus more heavily the distributed
serving setting for large models and oﬀer some incremental optimisations, but otherwise the core
underlying idea is the same.
Auto-regressive Sampling
Whilst transformers can be trained eﬃciently and in parallel on TPUs and GPUs, samples are typically
drawn auto-regressively (See algorithm 1). For most applications, auto-regressive sampling (ArS) is
highlymemorybandwidthboundandthuscannotmakeeﬀectiveuseofmodernacceleratorhardware
(Shazeer, 2019). A memory bound model call only generates a single token for every sequence in the
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper