title: Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
summary: Speculative decoding (SD) accelerates large language model inference by using
  a smaller draft model to predict multiple tokens, which are then verified in
  parallel by the larger target model. However, the limited capacity of the draft
  model often necessitates tree-based sampling to improve prediction accuracy,
  where multiple candidates are generated at each step. We identify a key
  limitation in this approach: the candidates at the same step are derived from
  the same representation, limiting diversity and reducing overall effectiveness.
  To address this, we propose Jakiro, leveraging Mixture of Experts (MoE), where
  independent experts generate diverse predictions, effectively decoupling
  correlations among candidates. Furthermore, we introduce a hybrid inference
  strategy, combining autoregressive decoding for initial tokens with parallel
  decoding for subsequent stages, and enhance the latter with contrastive
  mechanism in features to improve accuracy. Our method significantly boosts
  prediction accuracy and achieves higher inference speedups. Extensive
  experiments across diverse models validate the effectiveness and robustness of
  our approach, establishing a new SOTA in speculative decoding. Our codes are
  available at https://github.com/haiduo/Jakiro.
paper_authors._id: 67ab6c9867a1607ab478b975
paper_authors.name: Haiduo Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:33:50.773Z
paper_authors._id: 65780c60411e14898b8da93e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/fvjP6qICiuR09LV8Xzahb.png
paper_authors.fullname: Haiduo Huang
paper_authors.type: user
paper_authors.user: Hhaiduo
paper_authors._id: 67ab6c9867a1607ab478b976
paper_authors.name: Fuwei Yang
paper_authors._id: 67ab6c9867a1607ab478b977
paper_authors.name: Zhenhua Liu
paper_authors._id: 67ab6c9867a1607ab478b978
paper_authors.name: Yixing Xu
paper_authors._id: 67ab6c9867a1607ab478b979
paper_authors.name: Jinze Li
paper_authors._id: 67ab6c9867a1607ab478b97a
paper_authors.name: Yang Liu
paper_authors._id: 67ab6c9867a1607ab478b97b
paper_authors.name: Xuanwu Yin
paper_authors._id: 67ab6c9867a1607ab478b97c
paper_authors.name: Dong Li
paper_authors._id: 67ab6c9867a1607ab478b97d
paper_authors.name: Pengju Ren
paper_authors._id: 67ab6c9867a1607ab478b97e
paper_authors.name: Emad Barsoum
paper_title: Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
paper_summary: Speculative decoding (SD) accelerates large language model inference by using
  a smaller draft model to predict multiple tokens, which are then verified in
  parallel by the larger target model. However, the limited capacity of the draft
  model often necessitates tree-based sampling to improve prediction accuracy,
  where multiple candidates are generated at each step. We identify a key
  limitation in this approach: the candidates at the same step are derived from
  the same representation, limiting diversity and reducing overall effectiveness.
  To address this, we propose Jakiro, leveraging Mixture of Experts (MoE), where
  independent experts generate diverse predictions, effectively decoupling
  correlations among candidates. Furthermore, we introduce a hybrid inference
  strategy, combining autoregressive decoding for initial tokens with parallel
  decoding for subsequent stages, and enhance the latter with contrastive
  mechanism in features to improve accuracy. Our method significantly boosts
  prediction accuracy and achieves higher inference speedups. Extensive
  experiments across diverse models validate the effectiveness and robustness of
  our approach, establishing a new SOTA in speculative decoding. Our codes are
  available at https://github.com/haiduo/Jakiro.
paper_ai_summary: Jakiro, a speculative decoding method using Mixture of Experts, enhances prediction accuracy and inference speed by generating diverse candidates and combining autoregressive and parallel decoding strategies.
paper_ai_keywords: speculative decoding draft model target model tree-based sampling Mixture of Experts independent experts hybrid inference strategy autoregressive decoding parallel decoding contrastive mechanism