Introduction
Scaling transformer models to 500B+ parameters has led to large performance improvements on
many natural language, computer vision and reinforcement learning tasks (Arnab et al., 2021; Brown
etal.,2020;Chowdheryetal.,2022;Dosovitskiyetal.,2020;Hoﬀmannetal.,2022;Raeetal.,2021).
However, transformer decoding remains a highly costly and ineﬃcient process in this regime.
Transformer sampling is typically memory bandwidth bound (Shazeer, 2019), so for a given set of
hardware, the time to generate a single token in transformer models is proportional to a ﬁrst order
approximationtothesizeofparametersandthesizeofthetransformermemory. Thesizeoflanguage
models also necessitates serving with model parallelism – adding communication overheads (Pope
et al., 2022) and multiplying resource requirements. Since each new token depends on the past,
many such transformer calls are required to sample a new sequence.
We present an algorithm to accelerate transformer sampling for latency critical applications, which
we call speculative sampling (SpS). This is achieved by:
1.Generating a short draft of length 𝐾. This can be attained with either a parallel model (Stern
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper