Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Knowledge distillation can be a cost-effective technique to distill knowledge
in Large Language Models, if the teacher output logits can be pre-computed and
cached. However, successfully applying this to pre-training remains largely
unexplored. In this work, we prove that naive approaches for sparse knowledge
distillation such as caching Top-K probabilities, while intuitive, provide
biased estimates of teacher probability distribution to the student, resulting
in suboptimal performance and calibration. We propose an
importance-sampling-based method `Random Sampling Knowledge Distillation',
which provides unbiased estimates, preserves the gradient in expectation, and
requires storing significantly sparser logits. Our method enables faster
training of student models with marginal overhead (<10%) compared to
cross-entropy based training, while maintaining competitive performance
compared to full distillation, across a range of model sizes from 300M to 3B.
67e4c860136c8a867191e52e
Anshumann
admin_assigned
2025-03-27T10:03:45.930Z
6627ff2a3b4fbc8420a416c3
/avatars/de3aafdaf5563fe25edcdb92b394474f.svg
AR
user
Anshumann
67e4c860136c8a867191e52f
Mohd Abbas Zaidi
admin_assigned
2025-03-27T10:03:57.163Z
61765fe0b0715831eab6d465
/avatars/830406cebdd7ddf66799bf71f6bbff4b.svg
Mohd Abbas Zaidi
user
ya-mehdi
67e4c860136c8a867191e530
Akhil Kedia
67e4c860136c8a867191e531
Jinwoo Ahn
admin_assigned
2025-03-27T10:04:07.687Z
65d61eebee922bc7a777d5a6
/avatars/180efd503ef412b4dc728e6aa477c16e.svg
Jinwoo Ahn
user
AndrewAhn
67e4c860136c8a867191e532
Taehwak Kwon
admin_assigned
2025-03-27T10:04:23.731Z
629059cdb90dde28ef5cbb30
/avatars/451b0e8999447b3a2f03378fe98c0661.svg
Taehwak Kwon
user
Rock222
67e4c860136c8a867191e533
Kangwook Lee
67e4c860136c8a867191e534
Haejun Lee
admin_assigned
2025-03-27T10:04:39.155Z
653b6e7232bd4db35d981615
/avatars/0fbf0c5d502b840bf26baf8c420c7593.svg
Haejun Lee
user
haejunlee
67e4c860136c8a867191e535
Joohyung Lee
admin_assigned
2025-03-27T10:04:46.702Z
64b4be0665a7e15eac085878
/avatars/dd19ec2987fb0735457c6492b53aacfe.svg
Joo-hyung Lee
user
snrbs17
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Knowledge distillation can be a cost-effective technique to distill knowledge
in Large Language Models, if the teacher output logits can be pre-computed and
cached. However, successfully applying this to pre-training remains largely
unexplored. In this work, we prove that naive approaches for sparse knowledge
distillation such as caching Top-K probabilities, while intuitive, provide
biased estimates of teacher probability distribution to the student, resulting
in suboptimal performance and calibration. We propose an
importance-sampling-based method `Random Sampling Knowledge Distillation',
which provides unbiased estimates, preserves the gradient in expectation, and
requires storing significantly sparser logits. Our method enables faster
training of student models with marginal overhead (<10%) compared to
cross-entropy based training, while maintaining competitive performance
compared to full distillation, across a range of model sizes from 300M to 3B.
Random Sampling Knowledge Distillation provides unbiased and efficient knowledge transfer by sampling logits, enhancing training speed with minimal performance loss compared to full distillation.
sparse knowledge distillation knowledge distillation teacher output logits student model cross-entropy based training