length, potentially giving us a larger speedup. However, the total loop time increases approximately
linearly with the larger number of draft model calls and small increases in the scoring time. The
overall eﬃciency of the proportion of accepted tokens decreases as 𝐾increases, since later tokens
depend on the acceptance of previous tokens. This results in the average speedup plateauing or
even degrading with a larger 𝐾(for example, XSum with nucleus’s latency is minimised at 𝐾=3),
depending on the domain.
7
Accelerating Large Language Model Decoding with Speculative Sampling
Further, even though larger values of 𝐾may yield marginally greater mean speedups in certain
circumstances, it also increases variance of the time to generate a full sequence. This could be
problematic for settings where the P90, P99 latencies of concern.
Conclusion
Inthiswork,wedemonstrateanewalgorithmandworkﬂowforacceleratingthedecodingoflanguage
models. Speculative sampling does not require making any modiﬁcations to the target language
model’s parameters or architecture, is provably lossless within numerics, scales well with the appropriate draft model and complements many existing techniques for reducing latency in the small batch
size setting.
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper