title: DarwinLM: Evolutionary Structured Pruning of Large Language Models
summary: Large Language Models (LLMs) have achieved significant success across various
  NLP tasks. However, their massive computational costs limit their widespread
  use, particularly in real-time applications. Structured pruning offers an
  effective solution by compressing models and directly providing end-to-end
  speed improvements, regardless of the hardware environment. Meanwhile,
  different components of the model exhibit varying sensitivities towards
  pruning, calling for non-uniform model compression. However, a pruning
  method should not only identify a capable substructure, but also account for
  post-compression training. To this end, we propose \sysname, a method for
  training-aware structured pruning. \sysname builds upon an evolutionary
  search process, generating multiple offspring models in each generation through
  mutation, and selecting the fittest for survival. To assess the effect of
  post-training, we incorporate a lightweight, multistep training process within
  the offspring population, progressively increasing the number of tokens and
  eliminating poorly performing models in each selection stage. We validate our
  method through extensive experiments on Llama-2-7B, Llama-3.1-8B and
  Qwen-2.5-14B-Instruct, achieving state-of-the-art performance for structured
  pruning. For instance, \sysname surpasses ShearedLlama while requiring
  5times less training data during post-compression training.
paper_authors._id: 67b33f632f3994b7d95b6e77
paper_authors.name: Shengkun Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:22.646Z
paper_authors._id: 63e76e2bfdb4097ef65e0745
paper_authors.avatarUrl: /avatars/6d4d94ab6f44e23437488fd9fed2a383.svg
paper_authors.fullname: Tang
paper_authors.type: user
paper_authors.user: Shengkun
paper_authors._id: 67b33f632f3994b7d95b6e78
paper_authors.name: Oliver Sieberling
paper_authors._id: 67b33f632f3994b7d95b6e79
paper_authors.name: Eldar Kurtic
paper_authors._id: 67b33f632f3994b7d95b6e7a
paper_authors.name: Zhiqiang Shen
paper_authors._id: 67b33f632f3994b7d95b6e7b
paper_authors.name: Dan Alistarh
paper_title: DarwinLM: Evolutionary Structured Pruning of Large Language Models
paper_summary: Large Language Models (LLMs) have achieved significant success across various
  NLP tasks. However, their massive computational costs limit their widespread
  use, particularly in real-time applications. Structured pruning offers an
  effective solution by compressing models and directly providing end-to-end
  speed improvements, regardless of the hardware environment. Meanwhile,
  different components of the model exhibit varying sensitivities towards
  pruning, calling for non-uniform model compression. However, a pruning
  method should not only identify a capable substructure, but also account for
  post-compression training. To this end, we propose \sysname, a method for
  training-aware structured pruning. \sysname builds upon an evolutionary
  search process, generating multiple offspring models in each generation through
  mutation, and selecting the fittest for survival. To assess the effect of
  post-training, we incorporate a lightweight, multistep training process within
  the offspring population, progressively increasing the number of tokens and
  eliminating poorly performing models in each selection stage. We validate our
  method through extensive experiments on Llama-2-7B, Llama-3.1-8B and
  Qwen-2.5-14B-Instruct, achieving state-of-the-art performance for structured
  pruning. For instance, \sysname surpasses ShearedLlama while requiring
  5times less training data during post-compression training.
paper_ai_keywords: Large Language Models (LLMs) Structured pruning Non-uniform model compression Evolutionary search process Mutation Lightweight, multistep training process Tokens Post-training Offspring models Fittest for survival Training-aware structured pruning ShearedLlama