Sampling Method Benchmark Result Mean Token Time Speed Up
ArS (Nucleus)XSum (ROUGE-2)0.112 14.1ms/Token 1
SpS (Nucleus) 0.114 7.52ms/Token 192
ArS (Greedy)XSum (ROUGE-2)0.157 14.1ms/Token 1
SpS (Greedy) 0.156 7.00ms/Token 201
ArS (Nucleus)HumanEval (100 Shot)45.1% 14.1ms/Token 1
SpS (Nucleus) 47.0% 5.73ms/Token 246
on 16 TPU v4s (where it achieves the aforementioned 14.1ms/token ), whereas a chinchilla-optimal
7B achieves its lowest sampling latency on 4 TPU v4s (where it achieves 5ms/token ). For smaller
models, the additional memory bandwidth and ﬂops are insuﬃcient to oﬀset the additional communication overhead between more devices – serving a 7B on 16 TPUs actually increases the latency. This
means the 7B would provide only a modest speedup if used as a draft with its optimal topology, and
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper