Dynamic Depth Decoding: Faster Speculative Decoding for LLMs
<|reference_start|>Dynamic Depth Decoding: Faster Speculative Decoding for LLMs: The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x.<|reference_end|>
@article{brown2024dynamic,
    title={Dynamic Depth Decoding: Faster Speculative Decoding for LLMs},
    author={Oscar Brown, Zhengjie Wang, Andrea Do, Nikhil Mathew, Cheng Yu},
    journal={arXiv preprint arXiv:2409.00142},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.00142},
    primaryClass={cs.CL cs.AI}
}