post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861 , 2022.
Supplementary Materials
Author Contributions
•Initial proposal: Charlie Chen, John Jumper and Geoﬀrey Irving
•Initial Implementation, Optimisation and Scaling: Charlie Chen
•Modiﬁed Rejection Sampling Scheme: John Jumper
•Engineering Improvements: Jean-Baptiste Lespiau and Charlie Chen
•Experiments: Charlie Chen, Sebastian Borgeaud and Laurent Sifre
•Draft of Manuscript: Charlie Chen and Sebastian Borgeaud
•Manuscript Feedback: Laurent Sifre, Geoﬀrey Irving and John Jumper
Acknowledgements
We’d like to thank Oriol Vinyals and Koray Kavukcuoglu for your kind advice and leadership. We’d
also like to thank Evan Senter for your additional feedback on the manuscript and Amelia Glaese for
your support in navigating the publishing process. Finally, we’d like to thank Blake Hechtman, Berkin
Ilbeyi for your valuable advice on XLA and Nikolai Grigoriev for our discussions on the various tricks
that can be applied to the transformer architecture.
Hyperparams
Table 2jHyperparameters for the draft model
Model 𝑑modelHeads Layers Params
We present speculative sampling, an algorithm for accelerating transformer
decoding by enabling the generation of multiple tokens from each transformer
call. Our algorithm relies on the observation that the latency of parallel
scoring of short continuations, generated by a faster but less powerful draft
model, is comparable to that of sampling a single token from the larger target
model. This is combined with a novel modified rejection sampling scheme which
preserves the distribution of the target model within hardware numerics. We
benchmark speculative sampling with Chinchilla, a 70 billion parameter language
model, achieving a 2-2.5x decoding speedup in a distributed setup, without
compromising the sample quality or making modifications to the model itself.
Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre John Jumper