Total loop time
Figure 1jLeft:The average time to generate 128 tokens, with standard deviation. Note that as 𝐾
increases, the overall speedup plateaus or even regresses, with XSum being optimal at 𝐾=3. The
variance consistently increases with 𝐾.Middle:The average number of tokens accepted divided by
𝐾¸1– this serves as a measure of the overall eﬃciency of the modiﬁed rejection scheme, which
decreases with the lookahead. Right:Average time per loop increases approximately linearly with 𝐾
duetotheincreasednumberofmodelcalls. Notethatthegradientisslightlyhigherthanthesampling
speed of the draft model, due to additional overheads in nucleus decoding.
Trade oﬀ between longer drafts and more frequent scoring
Wevisualisethetrade-oﬀofincreasing 𝐾,thenumberoftokenssampledbythedraftmodelinFigure1.
As𝐾increases, we need fewer scoring calls from the large models to generate the same sequence
length, potentially giving us a larger speedup. However, the total loop time increases approximately
linearly with the larger number of draft model calls and small increases in the scoring time. The
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper