RecycleGPT: An Autoregressive Language Model with Recyclable Module
Existing large language models have to run K times to generate a sequence of
K tokens. In this paper, we present RecycleGPT, a generative language model
with fast decoding speed by recycling pre-generated model states without
running the whole model in multiple steps. Our approach relies on the
observation that adjacent tokens in a sequence usually have strong correlations
and the next token in a sequence can be reasonably guessed or inferred based on
the preceding ones. Through theoretical evaluations and practical tests on
downstream text generation tasks, we demonstrate the effectiveness of our
approach in lowering inference latency, achieving up to 1.4x speedup while
preserving high performance.
64d1acda73174cecdff78384
Yufan Jiang
admin_assigned
2023-08-08T15:11:28.639Z
64cb4699942890af934cc556
/avatars/67599ce6057a5a35fca3d6a1861cb1e2.svg
Jiangyufan
user
jiangyingjunn
64d1acda73174cecdff78385
Qiaozhi He
64d1acda73174cecdff78386
Xiaomin Zhuang
64d1acda73174cecdff78387
Zhihua Wu
64d1acda73174cecdff78388
Kunpeng Wang
64d1acda73174cecdff78389
Wenlai Zhao
64d1acda73174cecdff7838a
Guangwen Yang
RecycleGPT: An Autoregressive Language Model with Recyclable Module
Existing large language models have to run K times to generate a sequence of
K tokens. In this paper, we present RecycleGPT, a generative language model
with fast decoding speed by recycling pre-generated model states without
running the whole model in multiple steps. Our approach relies on the
observation that adjacent tokens in a sequence usually have strong correlations
and the next token in a sequence can be reasonably guessed or inferred based on
the preceding ones. Through theoretical evaluations and practical tests on
downstream text generation tasks, we demonstrate the effectiveness of our
approach in lowering inference latency, achieving up to 1.4x speedup while
preserving high performance.
RecycleGPT enhances language model decoding speed by reusing previous model states, reducing inference time with minimal performance loss.
generative language model fast decoding speed recycling pre-generated model states adjacent tokens strong correlations token inference theoretical evaluations downstream text generation tasks inference latency speedup