hardware that is typically used to serve Chinchilla for research purposes. This model was trained
with the same tokeniser and dataset as Chinchilla, with a slightly smaller width and with only 8
layers. The relatively few number of layers allows it to achieve a sampling speed of 1.8ms/token
comparedto 14.1ms/token forChinchilla. Fordetails,pleaserefertothehyperparametersinTable2.
For distributed setups it is insuﬃcient to naively choose a small model as the draft, since diﬀerent models have diﬀerent optimal inference setups. For example, it is typical to serve Chinchilla 70B
5
Accelerating Large Language Model Decoding with Speculative Sampling
Table1jChinchillaperformanceandspeedonXSumandHumanEvalwithnaiveandspeculative
sampling at batch size 1 and 𝐾=4. XSum was executed with nucleus parameter 𝑝=08, and
HumanEval with 𝑝=095and temperature 08.
Sampling Method Benchmark Result Mean Token Time Speed Up
ArS (Nucleus)XSum (ROUGE-2)0.112 14.1ms/Token 1
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper