𝑝¹˜𝑥𝑛¸1j𝑥1𝑥𝑛º
Where𝑞¹˜𝑥𝑛¸1j𝑥1𝑥𝑛ºand𝑝¹˜𝑥𝑛¸1j𝑥1𝑥𝑛ºare the probability of ˜𝑥𝑛¸1according to the target and
draft models respectively, conditioned on the context so far.
If the token is accepted, we set 𝑥𝑛¸1 ˜𝑥𝑛¸1and repeat the process for ˜𝑥𝑛¸2until either a token
is rejected or all tokens have been accepted.
If˜𝑥𝑛¸1is rejected, we resample 𝑥𝑛¸1from the following distribution:
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper