ReplaceMe: Network Simplification via Layer Pruning and Linear
  Transformations
We introduce ReplaceMe, a generalized training-free depth pruning method that
effectively replaces transformer blocks with a linear operation, while
maintaining high performance for low compression ratios. In contrast to
conventional pruning approaches that require additional training or
fine-tuning, our approach requires only a small calibration dataset that is
used to estimate a linear transformation to approximate the pruned blocks. This
estimated linear mapping can be seamlessly merged with the remaining
transformer blocks, eliminating the need for any additional network parameters.
Our experiments show that ReplaceMe consistently outperforms other
training-free approaches and remains highly competitive with state-of-the-art
pruning methods that involve extensive retraining/fine-tuning and architectural
modifications. Applied to several large language models (LLMs), ReplaceMe
achieves up to 25% pruning while retaining approximately 90% of the original
model's performance on open benchmarks - without any training or healing steps,
resulting in minimal computational overhead (see Fig.1). We provide an
open-source library implementing ReplaceMe alongside several state-of-the-art
depth pruning techniques, available at this repository.
6819b5da3d9c61444380f4c5
Dmitriy Shopkhoev
extracted_confirmed
2025-05-06T13:37:06.786Z
66465dfa508db0bde50d95f2
/avatars/8b4a583dc0f3cab0f1cd9a1be3daa01b.svg
Dmitry Shopkhoev
user
dimitriish
6819b5da3d9c61444380f4c6
Ammar Ali
claimed_verified
2025-05-06T08:32:17.870Z
6166db59f78a267701a78c2a
/avatars/8784efc36f67719e9455b1f081340ed9.svg
Ammar Ali
user
ammarali32
6819b5da3d9c61444380f4c7
Magauiya Zhussip
6819b5da3d9c61444380f4c8
Valentin Malykh
admin_assigned
2025-05-06T09:04:42.358Z
66b1ce4ca14db5aac3e5e755
/avatars/ab55ef112fba091813e1cc1f43857cf9.svg
Valentin Malykh
user
madrugado
6819b5da3d9c61444380f4c9
Stamatios Lefkimmiatis
claimed_verified
2025-05-06T08:32:13.923Z
6683cc62b466c0d8e60e1bbc
/avatars/d781cfb113263f88eaa3250bef521c53.svg
Stamatis Lefkimmiatis
user
stamatisl
6819b5da3d9c61444380f4ca
Nikos Komodakis
6819b5da3d9c61444380f4cb
Sergey Zagoruyko
admin_assigned
2025-05-06T09:04:52.244Z
667e7f968c6d7aede7ecb94b
/avatars/d6dabd9b909b1f20f661dc4bc07af23f.svg
Sergey Zagoruyko
user
szagoruyko121
ReplaceMe: Network Simplification via Layer Pruning and Linear
  Transformations
We introduce ReplaceMe, a generalized training-free depth pruning method that
effectively replaces transformer blocks with a linear operation, while
maintaining high performance for low compression ratios. In contrast to
conventional pruning approaches that require additional training or
fine-tuning, our approach requires only a small calibration dataset that is
used to estimate a linear transformation to approximate the pruned blocks. This
estimated linear mapping can be seamlessly merged with the remaining
transformer blocks, eliminating the need for any additional network parameters.
Our experiments show that ReplaceMe consistently outperforms other
training-free approaches and remains highly competitive with state-of-the-art
pruning methods that involve extensive retraining/fine-tuning and architectural
modifications. Applied to several large language models (LLMs), ReplaceMe
achieves up to 25% pruning while retaining approximately 90% of the original
model's performance on open benchmarks - without any training or healing steps,
resulting in minimal computational overhead (see Fig.1). We provide an
open-source library implementing ReplaceMe alongside several state-of-the-art
depth pruning techniques, available at this repository.
ReplaceMe is a training-free depth pruning method that replaces transformer blocks with linear operations, maintaining performance with minimal computational overhead.
transformer blocks depth pruning linear operation calibration dataset linear transformation merging parameter-efficient large language models LLMs open-source library