Accelerating Production LLMs with Combined Token/Embedding Speculators
<|reference_start|>Accelerating Production LLMs with Combined Token/Embedding Speculators: This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements.<|reference_end|>
@article{wertheimer2024accelerating,
    title={Accelerating Production LLMs with Combined Token/Embedding Speculators},
    author={Davis Wertheimer, Joshua Rosenkranz, Thomas Parnell, Sahil Suneja,
  Pavithra Ranganathan, Raghu Ganti, Mudhakar Srivatsa},
    journal={arXiv preprint arXiv:2404.19124},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.19124},
    primaryClass={cs.CL}
}