title: EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary
    Search
summary: The high computational costs of large language models (LLMs) have led to a
  flurry of research on LLM compression, via methods such as quantization,
  sparsification, or structured pruning. A new frontier in this area is given by
  dynamic, non-uniform compression methods, which adjust the compression
  levels (e.g., sparsity) per-block or even per-layer in order to minimize
  accuracy loss, while guaranteeing a global compression threshold. Yet, current
  methods rely on heuristics for identifying the "importance" of a given layer
  towards the loss, based on assumptions such as error monotonicity, i.e.
  that the end-to-end model compression error is proportional to the sum of
  layer-wise errors. In this paper, we revisit this area, and propose a new and
  general approach for dynamic compression that is provably optimal in a given
  input range. We begin from the motivating observation that, in general,
  error monotonicity does not hold for LLMs: compressed models with lower
  sum of per-layer errors can perform worse than models with higher error
  sums. To address this, we propose a new general evolutionary framework for
  dynamic LLM compression called EvoPress, which has provable convergence, and
  low sample and evaluation complexity. We show that these theoretical guarantees
  lead to highly competitive practical performance for dynamic compression of
  Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art
  results across all compression approaches: structural pruning (block/layer
  dropping), unstructured sparsity, as well as quantization with dynamic
  bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.
paper_authors._id: 671636703928a873ef375c78
paper_authors.name: Oliver Sieberling
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-21T11:10:17.633Z
paper_authors._id: 659ddfb45673a33b5db22d57
paper_authors.avatarUrl: /avatars/ae1dce603b4cae2659d6070e8ce98b15.svg
paper_authors.fullname: Oliver Sieberling
paper_authors.type: user
paper_authors.user: OliverSieberling
paper_authors._id: 671636703928a873ef375c79
paper_authors.name: Denis Kuznedelev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:53.847Z
paper_authors._id: 629cf0475a13ba8233dd18c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654452258405-noauth.jpeg
paper_authors.fullname: Denis Kuznedelev
paper_authors.type: user
paper_authors.user: SpiridonSunRotator
paper_authors._id: 671636703928a873ef375c7a
paper_authors.name: Eldar Kurtic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:59.511Z
paper_authors._id: 628e0ce4e53bbd334577fcb0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668801593252-628e0ce4e53bbd334577fcb0.jpeg
paper_authors.fullname: Eldar Kurtic
paper_authors.type: user
paper_authors.user: ekurtic
paper_authors._id: 671636703928a873ef375c7b
paper_authors.name: Dan Alistarh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:01:04.576Z
paper_authors._id: 64d100c5d8d0927372e3d4c0
paper_authors.avatarUrl: /avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
paper_authors.fullname: Dan Alistarh
paper_authors.type: user
paper_authors.user: dalistarh
paper_title: EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary
    Search
paper_summary: The high computational costs of large language models (LLMs) have led to a
  flurry of research on LLM compression, via methods such as quantization,
  sparsification, or structured pruning. A new frontier in this area is given by
  dynamic, non-uniform compression methods, which adjust the compression
  levels (e.g., sparsity) per-block or even per-layer in order to minimize
  accuracy loss, while guaranteeing a global compression threshold. Yet, current
  methods rely on heuristics for identifying the "importance" of a given layer
  towards the loss, based on assumptions such as error monotonicity, i.e.
  that the end-to-end model compression error is proportional to the sum of
  layer-wise errors. In this paper, we revisit this area, and propose a new and
  general approach for dynamic compression that is provably optimal in a given
  input range. We begin from the motivating observation that, in general,
  error monotonicity does not hold for LLMs: compressed models with lower
  sum of per-layer errors can perform worse than models with higher error
  sums. To address this, we propose a new general evolutionary framework for
  dynamic LLM compression called EvoPress, which has provable convergence, and
  low sample and evaluation complexity. We show that these theoretical guarantees
  lead to highly competitive practical performance for dynamic compression of
  Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art
  results across all compression approaches: structural pruning (block/layer
  dropping), unstructured sparsity, as well as quantization with dynamic
  bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.
paper_ai_summary: EvoPress, a new evolutionary framework, achieves state-of-the-art compression for large language models by addressing non-monotonic error in layer-wise compression.
paper_ai_keywords: quantization sparsification structured pruning dynamic compression non-uniform compression error monotonicity EvoPress LLM compression LLaMA Mistral Phi models structural pruning block dropping layer dropping unstructured sparsity quantization with dynamic bitwidths