title: Knowledge Distillation of Large Language Models
summary: Knowledge Distillation (KD) is a promising technique for reducing the high
  computational demand of large language models (LLMs). However, previous KD
  methods are primarily applied to white-box classification models or training
  small models to imitate black-box model APIs like ChatGPT. How to effectively
  distill the knowledge from white-box generative LLMs is still under-explored,
  which becomes more and more important with the prosperity of LLMs. In this
  work, we propose MiniLLM that distills smaller language models from generative
  larger language models. We first replace the forward Kullback-Leibler
  divergence (KLD) objective in the standard KD approaches with reverse KLD,
  which is more suitable for KD on generative language models, to prevent the
  student model from overestimating the low-probability regions of the teacher
  distribution. Then, we derive an effective optimization approach to learn this
  objective. Extensive experiments in the instruction-following setting show that
  the MiniLLM models generate more precise responses with the higher overall
  quality, lower exposure bias, better calibration, and higher long-text
  generation performance. Our method is also scalable for different model
  families with 120M to 13B parameters. We will release our code and model
  checkpoints at https://aka.ms/MiniLLM.
paper_authors._id: 648bc4a8d60140c4bbca5278
paper_authors.name: Yuxian Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T17:38:47.105Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 648bc4a8d60140c4bbca5279
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:20:13.208Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 648bc4a8d60140c4bbca527a
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T12:03:25.107Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 648bc4a8d60140c4bbca527b
paper_authors.name: Minlie Huang
paper_title: Knowledge Distillation of Large Language Models
paper_summary: Knowledge Distillation (KD) is a promising technique for reducing the high
  computational demand of large language models (LLMs). However, previous KD
  methods are primarily applied to white-box classification models or training
  small models to imitate black-box model APIs like ChatGPT. How to effectively
  distill the knowledge from white-box generative LLMs is still under-explored,
  which becomes more and more important with the prosperity of LLMs. In this
  work, we propose MiniLLM that distills smaller language models from generative
  larger language models. We first replace the forward Kullback-Leibler
  divergence (KLD) objective in the standard KD approaches with reverse KLD,
  which is more suitable for KD on generative language models, to prevent the
  student model from overestimating the low-probability regions of the teacher
  distribution. Then, we derive an effective optimization approach to learn this
  objective. Extensive experiments in the instruction-following setting show that
  the MiniLLM models generate more precise responses with the higher overall
  quality, lower exposure bias, better calibration, and higher long-text
  generation performance. Our method is also scalable for different model
  families with 120M to 13B parameters. We will release our code and model
  checkpoints at https://aka.ms/MiniLLM.
paper_ai_keywords: Knowledge Distillation computational demand large language models white-box generative LLMs MiniLLM forward Kullback-Leibler divergence reverse KULLback-Leibler divergence teacher distribution student model instruction-following exposure bias calibration long-text generation model parameters model checkpoints