PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
<|reference_start|>PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation: Inference of Large Language Models (LLMs) across computer clusters has become a focal point of research in recent times, with many acceleration techniques taking inspiration from CPU speculative execution. These techniques reduce bottlenecks associated with memory bandwidth, but also increase end-to-end latency per inference run, requiring high speculation acceptance rates to improve performance. Combined with a variable rate of acceptance across tasks, speculative inference techniques can result in reduced performance. Additionally, pipeline-parallel designs require many user requests to maintain maximum utilization. As a remedy, we propose PipeInfer, a pipelined speculative acceleration technique to reduce inter-token latency and improve system utilization for single-request scenarios while also improving tolerance to low speculation acceptance rates and low-bandwidth interconnects. PipeInfer exhibits up to a 2.15$\times$ improvement in generation speed over standard speculative inference. PipeInfer achieves its improvement through Continuous Asynchronous Speculation and Early Inference Cancellation, the former improving latency and generation speed by running single-token inference simultaneously with several speculative runs, while the latter improves speed and latency by skipping the computation of invalidated runs, even in the middle of inference.<|reference_end|>
@article{butler2024pipeinfer:,
    title={PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined
  Speculation},
    author={Branden Butler, Sixing Yu, Arya Mazaheri, and Ali Jannesari},
    journal={arXiv preprint arXiv:2407.11798},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.11798},
    primaryClass={cs.CL cs.DC cs.LG}
}