CLaSp: In-Context Layer Skip for Self-Speculative Decoding
Speculative decoding (SD) is a promising method for accelerating the decoding
process of Large Language Models (LLMs). The efficiency of SD primarily hinges
on the consistency between the draft model and the verify model. However,
existing drafting approaches typically require additional modules to be
trained, which can be challenging to implement and ensure compatibility across
various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping
strategy for self-speculative decoding. Unlike prior methods, CLaSp does not
require additional drafting modules or extra training. Instead, it employs a
plug-and-play mechanism by skipping intermediate layers of the verify model to
construct a compressed draft model. Specifically, we develop a dynamic
programming algorithm that optimizes the layer-skipping process by leveraging
the complete hidden states from the last verification stage as an objective.
This enables CLaSp to dynamically adjust its layer-skipping strategy after each
verification stage, without relying on pre-optimized sets of skipped layers.
Experimental results across diverse downstream tasks demonstrate that CLaSp
achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the
original distribution of the generated text.
683d29da83edd521f116444c
Longze Chen
claimed_verified
2025-06-04T09:03:19.668Z
64c7b4d1c547ed5243c07b6c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
Longze Chen
user
lzchen2001
683d29da83edd521f116444d
Renke Shan
683d29da83edd521f116444e
Huiming Wang
683d29da83edd521f116444f
Lu Wang
claimed_verified
2025-06-02T13:39:48.296Z
67ffb89774193e65a77cd914
/avatars/4302f06bcb1fa4c372b4d675459c4068.svg
Wang Lu
user
lwzzfzl
683d29da83edd521f1164450
Ziqiang Liu
683d29da83edd521f1164451
Run Luo
683d29da83edd521f1164452
Jiawei Wang
683d29da83edd521f1164453
Hamid Alinejad-Rokny
683d29da83edd521f1164454
Min Yang
CLaSp: In-Context Layer Skip for Self-Speculative Decoding
Speculative decoding (SD) is a promising method for accelerating the decoding
process of Large Language Models (LLMs). The efficiency of SD primarily hinges
on the consistency between the draft model and the verify model. However,
existing drafting approaches typically require additional modules to be
trained, which can be challenging to implement and ensure compatibility across
various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping
strategy for self-speculative decoding. Unlike prior methods, CLaSp does not
require additional drafting modules or extra training. Instead, it employs a
plug-and-play mechanism by skipping intermediate layers of the verify model to
construct a compressed draft model. Specifically, we develop a dynamic
programming algorithm that optimizes the layer-skipping process by leveraging
the complete hidden states from the last verification stage as an objective.
This enables CLaSp to dynamically adjust its layer-skipping strategy after each
verification stage, without relying on pre-optimized sets of skipped layers.
Experimental results across diverse downstream tasks demonstrate that CLaSp
achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the
original distribution of the generated text.
CLaSp, an in-context layer-skipping strategy for self-speculative decoding, accelerates Large Language Model decoding without additional modules or training, achieving a 1.3x to 1.7x speedup on LLaMA3 models.
speculative decoding large language models draft model verify model in-context layer-skipping dynamic programming algorithm hidden states verification stage