title: RecycleGPT: An Autoregressive Language Model with Recyclable Module
summary: Existing large language models have to run K times to generate a sequence of
  K tokens. In this paper, we present RecycleGPT, a generative language model
  with fast decoding speed by recycling pre-generated model states without
  running the whole model in multiple steps. Our approach relies on the
  observation that adjacent tokens in a sequence usually have strong correlations
  and the next token in a sequence can be reasonably guessed or inferred based on
  the preceding ones. Through theoretical evaluations and practical tests on
  downstream text generation tasks, we demonstrate the effectiveness of our
  approach in lowering inference latency, achieving up to 1.4x speedup while
  preserving high performance.
paper_authors._id: 64d1acda73174cecdff78384
paper_authors.name: Yufan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:11:28.639Z
paper_authors._id: 64cb4699942890af934cc556
paper_authors.avatarUrl: /avatars/67599ce6057a5a35fca3d6a1861cb1e2.svg
paper_authors.fullname: Jiangyufan
paper_authors.type: user
paper_authors.user: jiangyingjunn
paper_authors._id: 64d1acda73174cecdff78385
paper_authors.name: Qiaozhi He
paper_authors._id: 64d1acda73174cecdff78386
paper_authors.name: Xiaomin Zhuang
paper_authors._id: 64d1acda73174cecdff78387
paper_authors.name: Zhihua Wu
paper_authors._id: 64d1acda73174cecdff78388
paper_authors.name: Kunpeng Wang
paper_authors._id: 64d1acda73174cecdff78389
paper_authors.name: Wenlai Zhao
paper_authors._id: 64d1acda73174cecdff7838a
paper_authors.name: Guangwen Yang
paper_title: RecycleGPT: An Autoregressive Language Model with Recyclable Module
paper_summary: Existing large language models have to run K times to generate a sequence of
  K tokens. In this paper, we present RecycleGPT, a generative language model
  with fast decoding speed by recycling pre-generated model states without
  running the whole model in multiple steps. Our approach relies on the
  observation that adjacent tokens in a sequence usually have strong correlations
  and the next token in a sequence can be reasonably guessed or inferred based on
  the preceding ones. Through theoretical evaluations and practical tests on
  downstream text generation tasks, we demonstrate the effectiveness of our
  approach in lowering inference latency, achieving up to 1.4x speedup while
  preserving high performance.
paper_ai_keywords: generative language model fast decoding speed pre-generated model states correlations inference latency