Hierarchical Skip Decoding for Efficient Autoregressive Text Generation
<|reference_start|>Hierarchical Skip Decoding for Efficient Autoregressive Text Generation: Autoregressive decoding strategy is a commonly used method for text generation tasks with pre-trained language models, while early-exiting is an effective approach to speedup the inference stage. In this work, we propose a novel decoding strategy named Hierarchical Skip Decoding (HSD) for efficient autoregressive text generation. Different from existing methods that require additional trainable components, HSD is a plug-and-play method applicable to autoregressive text generation models, it adaptively skips decoding layers in a hierarchical manner based on the current sequence length, thereby reducing computational workload and allocating computation resources. Comprehensive experiments on five text generation datasets with pre-trained language models demonstrate HSD's advantages in balancing efficiency and text quality. With almost half of the layers skipped, HSD can sustain 90% of the text quality compared to vanilla autoregressive decoding, outperforming the competitive approaches.<|reference_end|>
@article{zhu2024hierarchical,
    title={Hierarchical Skip Decoding for Efficient Autoregressive Text Generation},
    author={Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Wensheng Zhang},
    journal={arXiv preprint arXiv:2403.14919},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14919},
    primaryClass={cs.CL cs.AI}
}