title: CLaSp: In-Context Layer Skip for Self-Speculative Decoding
summary: Speculative decoding (SD) is a promising method for accelerating the decoding
  process of Large Language Models (LLMs). The efficiency of SD primarily hinges
  on the consistency between the draft model and the verify model. However,
  existing drafting approaches typically require additional modules to be
  trained, which can be challenging to implement and ensure compatibility across
  various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping
  strategy for self-speculative decoding. Unlike prior methods, CLaSp does not
  require additional drafting modules or extra training. Instead, it employs a
  plug-and-play mechanism by skipping intermediate layers of the verify model to
  construct a compressed draft model. Specifically, we develop a dynamic
  programming algorithm that optimizes the layer-skipping process by leveraging
  the complete hidden states from the last verification stage as an objective.
  This enables CLaSp to dynamically adjust its layer-skipping strategy after each
  verification stage, without relying on pre-optimized sets of skipped layers.
  Experimental results across diverse downstream tasks demonstrate that CLaSp
  achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the
  original distribution of the generated text.
paper_authors._id: 683d29da83edd521f116444c
paper_authors.name: Longze Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:19.668Z
paper_authors._id: 64c7b4d1c547ed5243c07b6c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
paper_authors.fullname: Longze Chen
paper_authors.type: user
paper_authors.user: lzchen2001
paper_authors._id: 683d29da83edd521f116444d
paper_authors.name: Renke Shan
paper_authors._id: 683d29da83edd521f116444e
paper_authors.name: Huiming Wang
paper_authors._id: 683d29da83edd521f116444f
paper_authors.name: Lu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T13:39:48.296Z
paper_authors._id: 67ffb89774193e65a77cd914
paper_authors.avatarUrl: /avatars/4302f06bcb1fa4c372b4d675459c4068.svg
paper_authors.fullname: Wang Lu
paper_authors.type: user
paper_authors.user: lwzzfzl
paper_authors._id: 683d29da83edd521f1164450
paper_authors.name: Ziqiang Liu
paper_authors._id: 683d29da83edd521f1164451
paper_authors.name: Run Luo
paper_authors._id: 683d29da83edd521f1164452
paper_authors.name: Jiawei Wang
paper_authors._id: 683d29da83edd521f1164453
paper_authors.name: Hamid Alinejad-Rokny
paper_authors._id: 683d29da83edd521f1164454
paper_authors.name: Min Yang
paper_title: CLaSp: In-Context Layer Skip for Self-Speculative Decoding
paper_summary: Speculative decoding (SD) is a promising method for accelerating the decoding
  process of Large Language Models (LLMs). The efficiency of SD primarily hinges
  on the consistency between the draft model and the verify model. However,
  existing drafting approaches typically require additional modules to be
  trained, which can be challenging to implement and ensure compatibility across
  various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping
  strategy for self-speculative decoding. Unlike prior methods, CLaSp does not
  require additional drafting modules or extra training. Instead, it employs a
  plug-and-play mechanism by skipping intermediate layers of the verify model to
  construct a compressed draft model. Specifically, we develop a dynamic
  programming algorithm that optimizes the layer-skipping process by leveraging
  the complete hidden states from the last verification stage as an objective.
  This enables CLaSp to dynamically adjust its layer-skipping strategy after each
  verification stage, without relying on pre-optimized sets of skipped layers.
  Experimental results across diverse downstream tasks demonstrate that CLaSp
  achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the
  original distribution of the generated text.
paper_ai_summary: CLaSp, an in-context layer-skipping strategy for self-speculative decoding, accelerates Large Language Model decoding without additional modules or training, achieving a 1.3x to 1.7x speedup on LLaMA3 models.
paper_ai_keywords: speculative decoding large language models draft model verify model in-context layer-skipping dynamic programming algorithm hidden states verification stage