For the ﬁrst term, we apply the acceptance rule:
ℙ¹˜𝑥=𝑥ºℙ¹˜𝑥acceptedj˜𝑥=𝑥º
=𝑝¹𝑥ºmin
1𝑞¹𝑥º
𝑝¹𝑥º
10
Accelerating Large Language Model Decoding with Speculative Sampling
=min¹𝑝¹𝑥º𝑞¹𝑥ºº
For the second conditional term, we apply the resampling rule:
ℙ¹𝑋=𝑥j˜𝑥rejectedº=¹𝑞¹𝑥º𝑝¹𝑥ºº¸
Where¹º¸denotes:
¹𝑓¹𝑥ºº¸=max¹0 𝑓¹𝑥ººÍ
𝑥max¹0 𝑓¹𝑥ºº
Finally, we calculate the probability of rejection:
ℙ¹˜𝑥rejectedº=1ℙ¹˜𝑥acceptedº
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper