title: LLM Pruning and Distillation in Practice: The Minitron Approach
summary: We present a comprehensive report on compressing the Llama 3.1 8B and Mistral
  NeMo 12B models to 4B and 8B parameters, respectively, using pruning and
  distillation. We explore two distinct pruning strategies: (1) depth pruning and
  (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on
  common benchmarks from the LM Evaluation Harness. The models are then aligned
  with NeMo Aligner and tested in instruct-tuned versions. This approach produces
  a compelling 4B model from Llama 3.1 8B and a state-of-the-art
  Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo
  12B. We found that with no access to the original data, it is beneficial to
  slightly fine-tune teacher models on the distillation dataset. We open-source
  our base model weights on Hugging Face with a permissive license.
paper_authors._id: 66c6c723e8e0c0e0a2cafd1b
paper_authors.name: Sharath Turuvekere Sreenivas
paper_authors._id: 66c6c723e8e0c0e0a2cafd1c
paper_authors.name: Saurav Muralidharan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:20.700Z
paper_authors._id: 64caca9b4ca74090b97a8795
paper_authors.avatarUrl: /avatars/310f7703ff3c07d283c02aca56fe86df.svg
paper_authors.fullname: Saurav Muralidharan
paper_authors.type: user
paper_authors.user: srvm
paper_authors._id: 66c6c723e8e0c0e0a2cafd1d
paper_authors.name: Raviraj Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T16:28:37.622Z
paper_authors._id: 64b803eb17681d64b1ba446c
paper_authors.avatarUrl: /avatars/f49560794044802defd3421d6539a7dc.svg
paper_authors.fullname: Raviraj Joshi
paper_authors.type: user
paper_authors.user: raviraj-fk
paper_authors._id: 66c6c723e8e0c0e0a2cafd1e
paper_authors.name: Marcin Chochowski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T16:28:45.919Z
paper_authors._id: 617a4ee4941993035fbaf294
paper_authors.avatarUrl: /avatars/b5d0b14f7cc53dd7f4f7200228a6e342.svg
paper_authors.fullname: Marcin Chochowski
paper_authors.type: user
paper_authors.user: mchochowski
paper_authors._id: 66c6c723e8e0c0e0a2cafd1f
paper_authors.name: Mostofa Patwary
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T16:28:51.228Z
paper_authors._id: 630544b09d2531fabd156fd3
paper_authors.avatarUrl: /avatars/7b5374244a887577834fb4524ff76d01.svg
paper_authors.fullname: Mostofa Patwary
paper_authors.type: user
paper_authors.user: mpatwary
paper_authors._id: 66c6c723e8e0c0e0a2cafd20
paper_authors.name: Mohammad Shoeybi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T16:28:57.311Z
paper_authors._id: 6641544c695975af2cbd0da6
paper_authors.avatarUrl: /avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
paper_authors.fullname: Mohammad Shoeybi
paper_authors.type: user
paper_authors.user: shoeybi
paper_authors._id: 66c6c723e8e0c0e0a2cafd21
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T16:29:02.920Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 66c6c723e8e0c0e0a2cafd22
paper_authors.name: Jan Kautz
paper_authors._id: 66c6c723e8e0c0e0a2cafd23
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:47.182Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_title: LLM Pruning and Distillation in Practice: The Minitron Approach
paper_summary: We present a comprehensive report on compressing the Llama 3.1 8B and Mistral
  NeMo 12B models to 4B and 8B parameters, respectively, using pruning and
  distillation. We explore two distinct pruning strategies: (1) depth pruning and
  (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on
  common benchmarks from the LM Evaluation Harness. The models are then aligned
  with NeMo Aligner and tested in instruct-tuned versions. This approach produces
  a compelling 4B model from Llama 3.1 8B and a state-of-the-art
  Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo
  12B. We found that with no access to the original data, it is beneficial to
  slightly fine-tune teacher models on the distillation dataset. We open-source
  our base model weights on Hugging Face with a permissive license.
paper_ai_keywords: pruning depth pruning representation pruning hidden pruning attention pruning MLP pruning distillation LM Evaluation Harness NeMo Aligner instruct-tuned Text-to-Text Transfer Transformer (T5) distillation dataset fine-tuning Hugging Face