title: ReplaceMe: Network Simplification via Layer Pruning and Linear
    Transformations
summary: We introduce ReplaceMe, a generalized training-free depth pruning method that
  effectively replaces transformer blocks with a linear operation, while
  maintaining high performance for low compression ratios. In contrast to
  conventional pruning approaches that require additional training or
  fine-tuning, our approach requires only a small calibration dataset that is
  used to estimate a linear transformation to approximate the pruned blocks. This
  estimated linear mapping can be seamlessly merged with the remaining
  transformer blocks, eliminating the need for any additional network parameters.
  Our experiments show that ReplaceMe consistently outperforms other
  training-free approaches and remains highly competitive with state-of-the-art
  pruning methods that involve extensive retraining/fine-tuning and architectural
  modifications. Applied to several large language models (LLMs), ReplaceMe
  achieves up to 25% pruning while retaining approximately 90% of the original
  model's performance on open benchmarks - without any training or healing steps,
  resulting in minimal computational overhead (see Fig.1). We provide an
  open-source library implementing ReplaceMe alongside several state-of-the-art
  depth pruning techniques, available at this repository.
paper_authors._id: 6819b5da3d9c61444380f4c5
paper_authors.name: Dmitriy Shopkhoev
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-06T13:37:06.786Z
paper_authors._id: 66465dfa508db0bde50d95f2
paper_authors.avatarUrl: /avatars/8b4a583dc0f3cab0f1cd9a1be3daa01b.svg
paper_authors.fullname: Dmitry Shopkhoev
paper_authors.type: user
paper_authors.user: dimitriish
paper_authors._id: 6819b5da3d9c61444380f4c6
paper_authors.name: Ammar Ali
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:32:17.870Z
paper_authors._id: 6166db59f78a267701a78c2a
paper_authors.avatarUrl: /avatars/8784efc36f67719e9455b1f081340ed9.svg
paper_authors.fullname: Ammar Ali
paper_authors.type: user
paper_authors.user: ammarali32
paper_authors._id: 6819b5da3d9c61444380f4c7
paper_authors.name: Magauiya Zhussip
paper_authors._id: 6819b5da3d9c61444380f4c8
paper_authors.name: Valentin Malykh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:04:42.358Z
paper_authors._id: 66b1ce4ca14db5aac3e5e755
paper_authors.avatarUrl: /avatars/ab55ef112fba091813e1cc1f43857cf9.svg
paper_authors.fullname: Valentin Malykh
paper_authors.type: user
paper_authors.user: madrugado
paper_authors._id: 6819b5da3d9c61444380f4c9
paper_authors.name: Stamatios Lefkimmiatis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:32:13.923Z
paper_authors._id: 6683cc62b466c0d8e60e1bbc
paper_authors.avatarUrl: /avatars/d781cfb113263f88eaa3250bef521c53.svg
paper_authors.fullname: Stamatis Lefkimmiatis
paper_authors.type: user
paper_authors.user: stamatisl
paper_authors._id: 6819b5da3d9c61444380f4ca
paper_authors.name: Nikos Komodakis
paper_authors._id: 6819b5da3d9c61444380f4cb
paper_authors.name: Sergey Zagoruyko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:04:52.244Z
paper_authors._id: 667e7f968c6d7aede7ecb94b
paper_authors.avatarUrl: /avatars/d6dabd9b909b1f20f661dc4bc07af23f.svg
paper_authors.fullname: Sergey Zagoruyko
paper_authors.type: user
paper_authors.user: szagoruyko121
paper_title: ReplaceMe: Network Simplification via Layer Pruning and Linear
    Transformations
paper_summary: We introduce ReplaceMe, a generalized training-free depth pruning method that
  effectively replaces transformer blocks with a linear operation, while
  maintaining high performance for low compression ratios. In contrast to
  conventional pruning approaches that require additional training or
  fine-tuning, our approach requires only a small calibration dataset that is
  used to estimate a linear transformation to approximate the pruned blocks. This
  estimated linear mapping can be seamlessly merged with the remaining
  transformer blocks, eliminating the need for any additional network parameters.
  Our experiments show that ReplaceMe consistently outperforms other
  training-free approaches and remains highly competitive with state-of-the-art
  pruning methods that involve extensive retraining/fine-tuning and architectural
  modifications. Applied to several large language models (LLMs), ReplaceMe
  achieves up to 25% pruning while retaining approximately 90% of the original
  model's performance on open benchmarks - without any training or healing steps,
  resulting in minimal computational overhead (see Fig.1). We provide an
  open-source library implementing ReplaceMe alongside several state-of-the-art
  depth pruning techniques, available at this repository.
paper_ai_summary: ReplaceMe is a training-free depth pruning method that replaces transformer blocks with linear operations, maintaining performance with minimal computational overhead.
paper_ai_keywords: transformer blocks depth pruning linear operation calibration dataset linear transformation merging parameter-efficient large language models LLMs open-source library