Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
<|reference_start|>Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding: This research aims to accelerate the inference speed of large language models (LLMs) with billions of parameters. We propose \textbf{S}mart \textbf{P}arallel \textbf{A}uto-\textbf{C}orrect d\textbf{E}coding (SPACE), an innovative approach designed for achieving lossless acceleration of LLMs. By integrating semi-autoregressive inference and speculative decoding capabilities, SPACE uniquely enables autoregressive LLMs to parallelize token generation and verification. This is realized through a specialized semi-autoregressive supervised fine-tuning process that equips existing LLMs with the ability to simultaneously predict multiple tokens. Additionally, an auto-correct decoding algorithm facilitates the simultaneous generation and verification of token sequences within a single model invocation. Through extensive experiments on a range of LLMs, SPACE has demonstrated inference speedup ranging from 2.7x-4.0x on HumanEval-X while maintaining output quality.<|reference_end|>
@article{yi2024generation,
    title={Generation Meets Verification: Accelerating Large Language Model
  Inference with Smart Parallel Auto-Correct Decoding},
    author={Hanling Yi, Feng Lin, Hongbin Li, Peiyang Ning, Xiaotian Yu, Rong Xiao},
    journal={arXiv preprint arXiv:2402.11809},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11809},
    primaryClass={cs.CL cs.AI cs.LG}
}