title: Compact Language Models via Pruning and Knowledge Distillation
summary: Large language models (LLMs) targeting different deployment scales and sizes
  are currently produced by training each variant from scratch; this is extremely
  compute-intensive. In this paper, we investigate if pruning an existing LLM and
  then re-training it with a fraction (<3%) of the original training data can be
  a suitable alternative to repeated, full retraining. To this end, we develop a
  set of practical and effective compression best practices for LLMs that combine
  depth, width, attention and MLP pruning with knowledge distillation-based
  retraining; we arrive at these best practices through a detailed empirical
  exploration of pruning strategies for each axis, methods to combine axes,
  distillation strategies, and search techniques for arriving at optimal
  compressed architectures. We use this guide to compress the Nemotron-4 family
  of LLMs by a factor of 2-4x, and compare their performance to similarly-sized
  models on a variety of language modeling tasks. Deriving 8B and 4B models from
  an already pretrained 15B model using our approach requires up to 40x fewer
  training tokens per model compared to training from scratch; this results in
  compute cost savings of 1.8x for training the full model family (15B, 8B, and
  4B). Minitron models exhibit up to a 16% improvement in MMLU scores compared to
  training from scratch, perform comparably to other community models such as
  Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art
  compression techniques from the literature. We have open-sourced Minitron model
  weights on Huggingface, with corresponding supplementary material including
  example code available on GitHub.
paper_authors._id: 669ef80d65049e06488040ac
paper_authors.name: Saurav Muralidharan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:28:34.804Z
paper_authors._id: 64caca9b4ca74090b97a8795
paper_authors.avatarUrl: /avatars/310f7703ff3c07d283c02aca56fe86df.svg
paper_authors.fullname: Saurav Muralidharan
paper_authors.type: user
paper_authors.user: srvm
paper_authors._id: 669ef80d65049e06488040ad
paper_authors.name: Sharath Turuvekere Sreenivas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:28.676Z
paper_authors._id: 66a01db4d370743fd12531c0
paper_authors.avatarUrl: /avatars/e913dc3326058919f5179c567197e3af.svg
paper_authors.fullname: Sharath Turuvekere Sreenivas
paper_authors.type: user
paper_authors.user: sharathtsnv
paper_authors._id: 669ef80d65049e06488040ae
paper_authors.name: Raviraj Joshi
paper_authors._id: 669ef80d65049e06488040af
paper_authors.name: Marcin Chochowski
paper_authors._id: 669ef80d65049e06488040b0
paper_authors.name: Mostofa Patwary
paper_authors._id: 669ef80d65049e06488040b1
paper_authors.name: Mohammad Shoeybi
paper_authors._id: 669ef80d65049e06488040b2
paper_authors.name: Bryan Catanzaro
paper_authors._id: 669ef80d65049e06488040b3
paper_authors.name: Jan Kautz
paper_authors._id: 669ef80d65049e06488040b4
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:28:31.780Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_title: Compact Language Models via Pruning and Knowledge Distillation
paper_summary: Large language models (LLMs) targeting different deployment scales and sizes
  are currently produced by training each variant from scratch; this is extremely
  compute-intensive. In this paper, we investigate if pruning an existing LLM and
  then re-training it with a fraction (<3%) of the original training data can be
  a suitable alternative to repeated, full retraining. To this end, we develop a
  set of practical and effective compression best practices for LLMs that combine
  depth, width, attention and MLP pruning with knowledge distillation-based
  retraining; we arrive at these best practices through a detailed empirical
  exploration of pruning strategies for each axis, methods to combine axes,
  distillation strategies, and search techniques for arriving at optimal
  compressed architectures. We use this guide to compress the Nemotron-4 family
  of LLMs by a factor of 2-4x, and compare their performance to similarly-sized
  models on a variety of language modeling tasks. Deriving 8B and 4B models from
  an already pretrained 15B model using our approach requires up to 40x fewer
  training tokens per model compared to training from scratch; this results in
  compute cost savings of 1.8x for training the full model family (15B, 8B, and
  4B). Minitron models exhibit up to a 16% improvement in MMLU scores compared to
  training from scratch, perform comparably to other community models such as
  Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art
  compression techniques from the literature. We have open-sourced Minitron model
  weights on Huggingface, with corresponding supplementary material including
  example code available on GitHub.
paper_ai_summary: Compressing large language models through strategic pruning and knowledge distillation reduces training compute costs and achieves performance comparable or better than similarly sized models trained from scratch.
paper_ai_keywords: pruning depth pruning width pruning attention pruning MLP pruning knowledge distillation Nemotron-4 compressed architectures MMLU scores Minitron Mistral 7B Gemma 7B Llama-3 8B