(Shazeer, 2019). A memory bound model call only generates a single token for every sequence in the
batch, hence generating multiple tokens introduces a large amount of latency in any system which
makes use of it.
2
Accelerating Large Language Model Decoding with Speculative Sampling
Thisisespeciallyproblematicasthenumberofparameters inthemodelincreases. Sinceallthemodel
parameters need to pass through at least one accelerator chip, the model size divided by the total
memory bandwidth across all chips gives us a hard ceiling on the maximum auto-regressive sampling
speed. Larger models also require serving on multiple accelerators, introducing a further source of
latency due to inter-device communication overheads.
Algorithm 1 Auto-regressive (ArS) with Auto-Regressive Models
Givenauto-regressivetargetmodel 𝑞¹jºandinitialpromptsequence 𝑥1𝑥𝑡andtargetsequence
length𝑇.
Initialise𝑛 𝑡.
while𝑛 𝑇do
Sample𝑥𝑛¸1𝑞¹𝑥j𝑥1𝑥𝑛º
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper