LLM Pruning and Distillation in Practice: The Minitron Approach
<|reference_start|>LLM Pruning and Distillation in Practice: The Minitron Approach: We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo 12B. We found that with no access to the original data, it is beneficial to slightly fine-tune teacher models on the distillation dataset. We open-source our base model weights on Hugging Face with a permissive license.<|reference_end|>
@article{sreenivas2024llm,
    title={LLM Pruning and Distillation in Practice: The Minitron Approach},
    author={Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi,
  Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan
  Kautz, Pavlo Molchanov},
    journal={arXiv preprint arXiv:2408.11796},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.11796},
    primaryClass={cs.CL cs.AI cs.LG}
}