SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding
<|reference_start|>SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding: Large Language Models (LLMs) demonstrate remarkable emergent abilities across various tasks, yet fall short of complex reasoning and planning tasks. The tree-search-based reasoning methods address this by surpassing the capabilities of chain-of-thought prompting, encouraging exploration of intermediate steps. However, such methods introduce significant inference latency due to the systematic exploration and evaluation of multiple thought paths. This paper introduces SeeD, a novel and efficient inference framework to optimize runtime speed and GPU memory management concurrently. By employing a scheduled speculative execution, SeeD efficiently handles multiple iterations for the thought generation and the state evaluation, leveraging a rounds-scheduled strategy to manage draft model dispatching. Extensive experimental evaluations on three reasoning datasets demonstrate superior speedup performance of SeeD, providing a viable path for batched inference in training-free speculative decoding.<|reference_end|>
@article{wang2024seed:,
    title={SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative
  Decoding},
    author={Zhenglin Wang, Jialong Wu, Yilong Lai, Congzhi Zhang, Deyu Zhou},
    journal={arXiv preprint arXiv:2406.18200},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.18200},
    primaryClass={cs.CL}
}