Distillation Scaling Laws
We provide a distillation scaling law that estimates distilled model
performance based on a compute budget and its allocation between the student
and teacher. Our findings reduce the risks associated with using distillation
at scale; compute allocation for both the teacher and student models can now be
done to maximize student performance. We provide compute optimal distillation
recipes for when 1) a teacher exists, or 2) a teacher needs training. If many
students are to be distilled, or a teacher already exists, distillation
outperforms supervised pretraining until a compute level which grows
predictably with student size. If one student is to be distilled and a teacher
also needs training, supervised learning should be done instead. Additionally,
we provide insights across our large scale study of distillation, which
increase our understanding of distillation and inform experimental design.
67ad77f9cd8de299e5049c05
Dan Busbridge
claimed_verified
2025-02-13T20:35:56.843Z
64c3726f2a5eaefd000cdedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64c3726f2a5eaefd000cdedd/iwFifH1sWQy7agW3eTmNQ.png
Dan Busbridge
user
dbusbridge
67ad77f9cd8de299e5049c06
Amitis Shidani
claimed_verified
2025-02-18T09:34:18.456Z
65bbdf954d1795fa8ee6795e
/avatars/209a656248bf57b9e717ed628414e34b.svg
Amitis Shidani
user
AmitisShidani
67ad77f9cd8de299e5049c07
Floris Weers
67ad77f9cd8de299e5049c08
Jason Ramapuram
67ad77f9cd8de299e5049c09
Etai Littwin
67ad77f9cd8de299e5049c0a
Russ Webb
Distillation Scaling Laws
We provide a distillation scaling law that estimates distilled model
performance based on a compute budget and its allocation between the student
and teacher. Our findings reduce the risks associated with using distillation
at scale; compute allocation for both the teacher and student models can now be
done to maximize student performance. We provide compute optimal distillation
recipes for when 1) a teacher exists, or 2) a teacher needs training. If many
students are to be distilled, or a teacher already exists, distillation
outperforms supervised pretraining until a compute level which grows
predictably with student size. If one student is to be distilled and a teacher
also needs training, supervised learning should be done instead. Additionally,
we provide insights across our large scale study of distillation, which
increase our understanding of distillation and inform experimental design.
The study presents a scaling law to optimize compute allocation for model distillation, showing conditions under which distillation outperforms supervised pretraining.
distillation scaling law distilled model compute budget student model teacher model compute optimal distillation supervised pretraining large scale study