means the 7B would provide only a modest speedup if used as a draft with its optimal topology, and
we will not make full utilisation of the hardware during drafting.
We can sidestep this issue by training a wider model with a relatively few number of layers in
order to minimise communication overhead. It has been observed that the performance of language
models is relatively robust to changes in model aspect ratio (Levine et al., 2020), so this allows us
to serve a powerful draft model which can be sampled rapidly on the same hardware as the target
model.
Evaluation on XSum and HumanEval
We evaluate speculative sampling with Chinchilla on two tasks and summarize the results in Table 1:
•The XSum (Narayan et al., 2018) benchmark. This is a natural language summarisation task
using a 1-shot prompt where we sample a total of 11,305 sequences with a maximum sequence
length 128.
•The 100-shot HumanEval task (Chen et al., 2021). This is a code generation task involves the
generation of 16,400 samples with a maximum sequence length of 512.
Even with greedy sampling, a single token deviating due to numerics could result in two sequences
diverging wildly. Since pseudo-random seeds are processed diﬀerently between ArS and SpS, and
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper