,then
Set𝑥𝑛¸𝑡 ˜𝑥𝑡and𝑛 𝑛¸1.
else
sample𝑥𝑛¸𝑡¹𝑞¹𝑥j𝑥1𝑥𝑛¸𝑡1º𝑝¹𝑥j𝑥1𝑥𝑛¸𝑡1ºº¸and exit for loop.
end if
end for
If all tokens 𝑥𝑛¸1𝑥𝑛¸𝐾are accepted, sample extra token 𝑥𝑛¸𝐾¸1𝑞¹𝑥j𝑥1𝑥𝑛𝑥𝑛¸𝐾ºand
set𝑛 𝑛¸1.
end while
Speculative Sampling
Conditional Scoring
For speculative sampling (See algorithm 2), we ﬁrst make the observation that computing the logits
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper