Recurrent Drafter for Fast Speculative Decoding in Large Language Models
<|reference_start|>Recurrent Drafter for Fast Speculative Decoding in Large Language Models: We present Recurrent Drafter (ReDrafter), an advanced speculative decoding approach that achieves state-of-the-art speedup for large language models (LLMs) inference. The performance gains are driven by three key aspects: (1) leveraging a recurrent neural network (RNN) as the draft model conditioning on LLM's hidden states, (2) applying a dynamic tree attention algorithm over beam search results to eliminate duplicated prefixes in candidate sequences, and (3) training through knowledge distillation from the LLM. ReDrafter accelerates Vicuna inference in MT-Bench by up to 3.5x with a PyTorch implementation on Nvidia H100 GPUs. To demonstrate its practicality in production environments, we integrate ReDrafter into TensorRT-LLM, reaching up to 2.5x speedup on H100 GPUs. We also validated its effectiveness for on-device applications by implementing the approach in MLX and benchmarking performance on Metal GPUs in Apple Silicon chips, achieving up to 2.3x speedup.<|reference_end|>
@article{cheng2024recurrent,
    title={Recurrent Drafter for Fast Speculative Decoding in Large Language Models},
    author={Yunfei Cheng and Aonan Zhang and Xuanyu Zhang and Chong Wang and Yi
  Wang},
    journal={arXiv preprint arXiv:2403.09919},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.09919},
    primaryClass={cs.CL cs.LG}
}