Speculative decoding (SD) (Leviathan et al., 2023) is a method for accelerating the inference of auto-
regressive transformer models by employing a faster assistant model. The assistant model generates
a sequence of candidate tokens, all of which are verified by the main model in a single forward
pass. By generating with the faster assistant model and only performing validation forward passes
with the main model, the decoding process is sped-up significantly. The i-th candidate token from
the assistant model ˆyi is only kept if all previous candidate tokens ˆy<i match the validation tokens
predicted by the main model. Consequently, speculative decoding ensures that the generated output
exactly matches the sequence of tokens that would be generated by the main model, making it a
natural replacement for existing inference pipelines that use the main model.
DistilSpec (Zhou et al., 2023) proposes using a knowledge-distilled student model as the assistant to
DISTIL-WHISPER