title: Distillation Scaling Laws
summary: We provide a distillation scaling law that estimates distilled model
  performance based on a compute budget and its allocation between the student
  and teacher. Our findings reduce the risks associated with using distillation
  at scale; compute allocation for both the teacher and student models can now be
  done to maximize student performance. We provide compute optimal distillation
  recipes for when 1) a teacher exists, or 2) a teacher needs training. If many
  students are to be distilled, or a teacher already exists, distillation
  outperforms supervised pretraining until a compute level which grows
  predictably with student size. If one student is to be distilled and a teacher
  also needs training, supervised learning should be done instead. Additionally,
  we provide insights across our large scale study of distillation, which
  increase our understanding of distillation and inform experimental design.
paper_authors._id: 67ad77f9cd8de299e5049c05
paper_authors.name: Dan Busbridge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:35:56.843Z
paper_authors._id: 64c3726f2a5eaefd000cdedd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c3726f2a5eaefd000cdedd/iwFifH1sWQy7agW3eTmNQ.png
paper_authors.fullname: Dan Busbridge
paper_authors.type: user
paper_authors.user: dbusbridge
paper_authors._id: 67ad77f9cd8de299e5049c06
paper_authors.name: Amitis Shidani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:34:18.456Z
paper_authors._id: 65bbdf954d1795fa8ee6795e
paper_authors.avatarUrl: /avatars/209a656248bf57b9e717ed628414e34b.svg
paper_authors.fullname: Amitis Shidani
paper_authors.type: user
paper_authors.user: AmitisShidani
paper_authors._id: 67ad77f9cd8de299e5049c07
paper_authors.name: Floris Weers
paper_authors._id: 67ad77f9cd8de299e5049c08
paper_authors.name: Jason Ramapuram
paper_authors._id: 67ad77f9cd8de299e5049c09
paper_authors.name: Etai Littwin
paper_authors._id: 67ad77f9cd8de299e5049c0a
paper_authors.name: Russ Webb
paper_title: Distillation Scaling Laws
paper_summary: We provide a distillation scaling law that estimates distilled model
  performance based on a compute budget and its allocation between the student
  and teacher. Our findings reduce the risks associated with using distillation
  at scale; compute allocation for both the teacher and student models can now be
  done to maximize student performance. We provide compute optimal distillation
  recipes for when 1) a teacher exists, or 2) a teacher needs training. If many
  students are to be distilled, or a teacher already exists, distillation
  outperforms supervised pretraining until a compute level which grows
  predictably with student size. If one student is to be distilled and a teacher
  also needs training, supervised learning should be done instead. Additionally,
  we provide insights across our large scale study of distillation, which
  increase our understanding of distillation and inform experimental design.
paper_ai_keywords: distillation scaling law distilled model performance compute budget student teacher compute allocation compute optimal distillation recipes supervised pretraining