title: Ouroboros: Speculative Decoding with Large Model Enhanced Drafting
summary: Drafting-then-verifying decoding methods such as speculative decoding are
  widely adopted training-free methods to accelerate the inference of large
  language models (LLMs). Instead of employing an autoregressive process to
  decode tokens sequentially, speculative decoding initially creates drafts with
  an efficient small model. Then LLMs are required to conduct verification and
  correction in a non-autoregressive fashion to minimize time overhead.
  Generating longer drafts can lead to even more significant speedups once
  verified, but also incurs substantial trial and error costs if it fails.
  Suffering from the high verification failure probability, existing decoding
  methods cannot draft too much content for verification at one time, achieving
  sub-optimal inference acceleration. In this paper, we introduce Ouroboros,
  which constructs a phrase candidate pool from the verification process of LLMs
  to provide candidates for draft generation of the small model. Thereby,
  Ouroboros can further improve the efficiency and effectiveness of the initial
  drafts. The experimental results on typical text generation tasks show that
  Ouroboros achieves speedups of up to 1.9x and 2.8x compared to lookahead
  decoding and speculative decoding, respectively. The source code of Ouroboros
  is available at https://github.com/thunlp/Ouroboros.
paper_authors._id: 65d6cb3258e71c5328879180
paper_authors.name: Weilin Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:13:16.980Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 65d6cb3258e71c5328879181
paper_authors.name: Yuxiang Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:13:15.173Z
paper_authors._id: 65394f942ddab2beff1cc967
paper_authors.avatarUrl: /avatars/bdd5b47618758f6de080a51834a17655.svg
paper_authors.fullname: Shawn Huang
paper_authors.type: user
paper_authors.user: hyx21
paper_authors._id: 65d6cb3258e71c5328879182
paper_authors.name: Xu Han
paper_authors._id: 65d6cb3258e71c5328879183
paper_authors.name: Chaojun Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T06:57:49.844Z
paper_authors._id: 608f6d72283d0a8d7be9d1f9
paper_authors.avatarUrl: /avatars/7f499a37019359a3c488ba6cc11751fc.svg
paper_authors.fullname: Chaojun XIAO
paper_authors.type: user
paper_authors.user: xcjthu
paper_authors._id: 65d6cb3258e71c5328879184
paper_authors.name: Zhiyuan Liu
paper_authors._id: 65d6cb3258e71c5328879185
paper_authors.name: Maosong Sun
paper_title: Ouroboros: Speculative Decoding with Large Model Enhanced Drafting
paper_summary: Drafting-then-verifying decoding methods such as speculative decoding are
  widely adopted training-free methods to accelerate the inference of large
  language models (LLMs). Instead of employing an autoregressive process to
  decode tokens sequentially, speculative decoding initially creates drafts with
  an efficient small model. Then LLMs are required to conduct verification and
  correction in a non-autoregressive fashion to minimize time overhead.
  Generating longer drafts can lead to even more significant speedups once
  verified, but also incurs substantial trial and error costs if it fails.
  Suffering from the high verification failure probability, existing decoding
  methods cannot draft too much content for verification at one time, achieving
  sub-optimal inference acceleration. In this paper, we introduce Ouroboros,
  which constructs a phrase candidate pool from the verification process of LLMs
  to provide candidates for draft generation of the small model. Thereby,
  Ouroboros can further improve the efficiency and effectiveness of the initial
  drafts. The experimental results on typical text generation tasks show that
  Ouroboros achieves speedups of up to 1.9x and 2.8x compared to lookahead
  decoding and speculative decoding, respectively. The source code of Ouroboros
  is available at https://github.com/thunlp/Ouroboros.
paper_ai_keywords: speculative decoding decoding methods large language models (LLMs) autoregressive process verification non-autoregressive draft generation phrase candidate pool lookahead decoding text generation tasks