EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary
  Search
The high computational costs of large language models (LLMs) have led to a
flurry of research on LLM compression, via methods such as quantization,
sparsification, or structured pruning. A new frontier in this area is given by
dynamic, non-uniform compression methods, which adjust the compression
levels (e.g., sparsity) per-block or even per-layer in order to minimize
accuracy loss, while guaranteeing a global compression threshold. Yet, current
methods rely on heuristics for identifying the "importance" of a given layer
towards the loss, based on assumptions such as error monotonicity, i.e.
that the end-to-end model compression error is proportional to the sum of
layer-wise errors. In this paper, we revisit this area, and propose a new and
general approach for dynamic compression that is provably optimal in a given
input range. We begin from the motivating observation that, in general,
error monotonicity does not hold for LLMs: compressed models with lower
sum of per-layer errors can perform worse than models with higher error
sums. To address this, we propose a new general evolutionary framework for
dynamic LLM compression called EvoPress, which has provable convergence, and
low sample and evaluation complexity. We show that these theoretical guarantees
lead to highly competitive practical performance for dynamic compression of
Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art
results across all compression approaches: structural pruning (block/layer
dropping), unstructured sparsity, as well as quantization with dynamic
bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.
671636703928a873ef375c78
Oliver Sieberling
extracted_confirmed
2024-10-21T11:10:17.633Z
659ddfb45673a33b5db22d57
/avatars/ae1dce603b4cae2659d6070e8ce98b15.svg
Oliver Sieberling
user
OliverSieberling
671636703928a873ef375c79
Denis Kuznedelev
admin_assigned
2024-10-23T08:00:53.847Z
629cf0475a13ba8233dd18c9
https://cdn-avatars.huggingface.co/v1/production/uploads/1654452258405-noauth.jpeg
Denis Kuznedelev
user
SpiridonSunRotator
671636703928a873ef375c7a
Eldar Kurtic
admin_assigned
2024-10-23T08:00:59.511Z
628e0ce4e53bbd334577fcb0
https://cdn-avatars.huggingface.co/v1/production/uploads/1668801593252-628e0ce4e53bbd334577fcb0.jpeg
Eldar Kurtic
user
ekurtic
671636703928a873ef375c7b
Dan Alistarh
admin_assigned
2024-10-23T08:01:04.576Z
64d100c5d8d0927372e3d4c0
/avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
Dan Alistarh
user
dalistarh
EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary
  Search
The high computational costs of large language models (LLMs) have led to a
flurry of research on LLM compression, via methods such as quantization,
sparsification, or structured pruning. A new frontier in this area is given by
dynamic, non-uniform compression methods, which adjust the compression
levels (e.g., sparsity) per-block or even per-layer in order to minimize
accuracy loss, while guaranteeing a global compression threshold. Yet, current
methods rely on heuristics for identifying the "importance" of a given layer
towards the loss, based on assumptions such as error monotonicity, i.e.
that the end-to-end model compression error is proportional to the sum of
layer-wise errors. In this paper, we revisit this area, and propose a new and
general approach for dynamic compression that is provably optimal in a given
input range. We begin from the motivating observation that, in general,
error monotonicity does not hold for LLMs: compressed models with lower
sum of per-layer errors can perform worse than models with higher error
sums. To address this, we propose a new general evolutionary framework for
dynamic LLM compression called EvoPress, which has provable convergence, and
low sample and evaluation complexity. We show that these theoretical guarantees
lead to highly competitive practical performance for dynamic compression of
Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art
results across all compression approaches: structural pruning (block/layer
dropping), unstructured sparsity, as well as quantization with dynamic
bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.
EvoPress, a new evolutionary framework, achieves state-of-the-art compression for large language models by addressing non-monotonic error in layer-wise compression.
quantization sparsification structured pruning dynamic compression non-uniform compression error monotonicity EvoPress LLM compression LLaMA Mistral Phi models structural pruning block dropping layer dropping unstructured sparsity quantization with dynamic bitwidths