2023-2-3
Accelerating Large Language Model Decoding
with Speculative Sampling
Charlie Chen1, Sebastian Borgeaud1, Geoﬀrey Irving1, Jean-Baptiste Lespiau1, Laurent Sifre1and John
Jumper1
1All authors from DeepMind
We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the
generation of multiple tokens from each transformer call. Our algorithm relies on the observation that
the latency of parallel scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target model. This is combined
with a novel modiﬁed rejection sampling scheme which preserves the distribution of the target model
withinhardwarenumerics. WebenchmarkspeculativesamplingwithChinchilla,a70billionparameter
language model, achieving a 2–25decoding speedup in a distributed setup, without compromising
the sample quality or making modiﬁcations to the model itself.
Introduction
Scaling transformer models to 500B+ parameters has led to large performance improvements on
many natural language, computer vision and reinforcement learning tasks (Arnab et al., 2021; Brown
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper