case of HumanEval and greedy XSum, this speedup exceeded the theoretical memory bandwidth limit
of the hardware for autoregressive sampling (model size divided by the total memory bandwidth).
Acceptance rate changes per domain
It is apparent that the acceptance rate is dependent on the application and the decoding method.
HumanEvalachievesasigniﬁcantlylargerspeedup—Wehypothesizethatthisisduetoacombination
of code containing a lot of common sub-sequences (e.g. for i in range(len(arr)): would be
relatively easy for a draft model to guess), is often decomposed into a smaller set of shorter tokens
and the temperature value sharpening both the draft and target logits.
01234567
Number of draft tokens (K)40060080010001200140016001800ms
Mean Sampling Time (128 tokens)
Human Eval
XSum
01234567
Number of draft tokens (K)0.50.60.70.80.91.0
Acceptance rate
Human Eval
XSum
01234567
Number of draft tokens (K)1416182022242628ms
Total loop time
Figure 1jLeft:The average time to generate 128 tokens, with standard deviation. Note that as 𝐾
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper