title: DLP: Dynamic Layerwise Pruning in Large Language Models
summary: Pruning has recently been widely adopted to reduce the parameter scale and
  improve the inference efficiency of Large Language Models (LLMs). Mainstream
  pruning techniques often rely on uniform layerwise pruning strategies, which
  can lead to severe performance degradation at high sparsity levels. Recognizing
  the varying contributions of different layers in LLMs, recent studies have
  shifted their focus toward non-uniform layerwise pruning. However, these
  approaches often rely on pre-defined values, which can result in suboptimal
  performance. To overcome these limitations, we propose a novel method called
  Dynamic Layerwise Pruning (DLP). This approach adaptively determines the
  relative importance of each layer by integrating model weights with input
  activation information, assigning pruning rates accordingly. Experimental
  results show that DLP effectively preserves model performance at high sparsity
  levels across multiple LLMs. Specifically, at 70% sparsity, DLP reduces the
  perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7%
  compared to state-of-the-art methods. Moreover, DLP is compatible with various
  existing LLM compression techniques and can be seamlessly integrated into
  Parameter-Efficient Fine-Tuning (PEFT). We release the code at
  https://github.com/ironartisan/DLP to facilitate future research.
paper_authors._id: 683fec0a9f37285365be6142
paper_authors.name: Yuli Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T15:03:32.826Z
paper_authors._id: 656201912d309fa7e27ddf40
paper_authors.avatarUrl: /avatars/d1bb9b263a758a0b0e7f803f4f888e95.svg
paper_authors.fullname: Yuli chen
paper_authors.type: user
paper_authors.user: yulichen
paper_authors._id: 683fec0a9f37285365be6143
paper_authors.name: Bo Cheng
paper_authors._id: 683fec0a9f37285365be6144
paper_authors.name: Jiale Han
paper_authors._id: 683fec0a9f37285365be6145
paper_authors.name: Yingying Zhang
paper_authors._id: 683fec0a9f37285365be6146
paper_authors.name: Yingting Li
paper_authors._id: 683fec0a9f37285365be6147
paper_authors.name: Shuhao Zhang
paper_title: DLP: Dynamic Layerwise Pruning in Large Language Models
paper_summary: Pruning has recently been widely adopted to reduce the parameter scale and
  improve the inference efficiency of Large Language Models (LLMs). Mainstream
  pruning techniques often rely on uniform layerwise pruning strategies, which
  can lead to severe performance degradation at high sparsity levels. Recognizing
  the varying contributions of different layers in LLMs, recent studies have
  shifted their focus toward non-uniform layerwise pruning. However, these
  approaches often rely on pre-defined values, which can result in suboptimal
  performance. To overcome these limitations, we propose a novel method called
  Dynamic Layerwise Pruning (DLP). This approach adaptively determines the
  relative importance of each layer by integrating model weights with input
  activation information, assigning pruning rates accordingly. Experimental
  results show that DLP effectively preserves model performance at high sparsity
  levels across multiple LLMs. Specifically, at 70% sparsity, DLP reduces the
  perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7%
  compared to state-of-the-art methods. Moreover, DLP is compatible with various
  existing LLM compression techniques and can be seamlessly integrated into
  Parameter-Efficient Fine-Tuning (PEFT). We release the code at
  https://github.com/ironartisan/DLP to facilitate future research.
paper_ai_summary: A dynamic layerwise pruning method adaptively determines layer importance by combining model weights and activation information to maintain performance in large language models at high sparsity.
paper_ai_keywords: pruning Large Language Models (LLMs) uniform layerwise pruning non-uniform layerwise pruning Dynamic Layerwise Pruning (DLP) perplexity Parameter-Efficient Fine-Tuning (PEFT)