that can be applied to the transformer architecture.
Hyperparams
Table 2jHyperparameters for the draft model
Model 𝑑modelHeads Layers Params
Target (Chinchilla) 8192 64 80 70B
Draft 6144 48 8 4B
Proofs
Theorem1 (ModiﬁedRejectionSamplingrecoversthetargetdistribution) .Givendiscretedistributions
𝑞,𝑝and a single draft sample ˜𝑥𝑝, let𝑋be the ﬁnal resulting sample. For 𝑋=𝑥to be true, we must
either sample ˜𝑥=𝑥and then accept it, or resample it after ˜𝑥(of any value) is rejected. Hence:
ℙ¹𝑋=𝑥º
=ℙ¹˜𝑥=𝑥ºℙ¹˜𝑥acceptedj˜𝑥=𝑥º¸ℙ¹˜𝑥rejectedºℙ¹𝑋=𝑥j˜𝑥rejectedº
For the ﬁrst term, we apply the acceptance rule:
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper