title: The LLM Surgeon
summary: State-of-the-art language models are becoming increasingly large in an effort
  to achieve the highest performance on large corpora of available textual data.
  However, the sheer size of the Transformer architectures makes it difficult to
  deploy models within computational, environmental or device-specific
  constraints. We explore data-driven compression of existing pretrained models
  as an alternative to training smaller models from scratch. To do so, we scale
  Kronecker-factored curvature approximations of the target loss landscape to
  large language models. In doing so, we can compute both the dynamic allocation
  of structures that can be removed as well as updates of remaining weights that
  account for the removal. We provide a general framework for unstructured,
  semi-structured and structured pruning and improve upon weight updates to
  capture more correlations between weights, while remaining computationally
  efficient. Experimentally, our method can prune rows and columns from a range
  of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance,
  and achieve state-of-the-art results in unstructured and semi-structured
  pruning of large language models.
paper_authors._id: 658e398711f68f12eae6e23e
paper_authors.name: Tycho F. A. van der Ouderaa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-18T08:11:01.386Z
paper_authors._id: 662046a80e8d8b41b2aaeefb
paper_authors.avatarUrl: /avatars/45568fc404a6bc9bf9bc8b0b8ff5ba49.svg
paper_authors.fullname: Tycho van der Ouderaa
paper_authors.type: user
paper_authors.user: tychovdo
paper_authors._id: 658e398711f68f12eae6e23f
paper_authors.name: Markus Nagel
paper_authors._id: 658e398711f68f12eae6e240
paper_authors.name: Mart van Baalen
paper_authors._id: 658e398711f68f12eae6e241
paper_authors.name: Yuki M. Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:12:10.015Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_authors._id: 658e398711f68f12eae6e242
paper_authors.name: Tijmen Blankevoort
paper_title: The LLM Surgeon
paper_summary: State-of-the-art language models are becoming increasingly large in an effort
  to achieve the highest performance on large corpora of available textual data.
  However, the sheer size of the Transformer architectures makes it difficult to
  deploy models within computational, environmental or device-specific
  constraints. We explore data-driven compression of existing pretrained models
  as an alternative to training smaller models from scratch. To do so, we scale
  Kronecker-factored curvature approximations of the target loss landscape to
  large language models. In doing so, we can compute both the dynamic allocation
  of structures that can be removed as well as updates of remaining weights that
  account for the removal. We provide a general framework for unstructured,
  semi-structured and structured pruning and improve upon weight updates to
  capture more correlations between weights, while remaining computationally
  efficient. Experimentally, our method can prune rows and columns from a range
  of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance,
  and achieve state-of-the-art results in unstructured and semi-structured
  pruning of large language models.
paper_ai_summary: Data-driven compression using Kronecker-factored curvature approximations efficiently prunes large language models with minimal performance impact.
paper_ai_keywords: Kronecker-factored curvature approximations target loss landscape unstructured pruning semi-structured pruning structured pruning weight updates OPT models Llamav2-7B