diverging wildly. Since pseudo-random seeds are processed diﬀerently between ArS and SpS, and
because the diﬀerent computation graphs lead to diﬀerent numerics, we cannot not expect identical
outputs. However, we expect the samples to come from the same distribution within numerics and
we empirically verify this by evaluating these benchmarks.
We run the tasks at batch size 1 with SpS and ArS. The time taken per SpS/ArS loop has low
variance, and we can measure it directly from TPU proﬁles. To obtain the average speedup, standard
deviations and other metrics, we log the amount of tokens generated for each speculative loop. In
Table 1 we show the performance on the XSum and HumanEval benchmarks for naive and speculative
sampling with Chinchilla.
6
Accelerating Large Language Model Decoding with Speculative Sampling
We obtain a substantial speedup in both tasks, with HumanEval reaching speedups of almost 25.
Yet, we have parity in the benchmark metrics – the underlying samples distribution is provably the
sameuptonumerics, andthisveriﬁesthatthedraftmodelisnotbiasingtheresultsempirically. Inthe
case of HumanEval and greedy XSum, this speedup exceeded the theoretical memory bandwidth limit
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper