title: Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
summary: Knowledge distillation can be a cost-effective technique to distill knowledge
  in Large Language Models, if the teacher output logits can be pre-computed and
  cached. However, successfully applying this to pre-training remains largely
  unexplored. In this work, we prove that naive approaches for sparse knowledge
  distillation such as caching Top-K probabilities, while intuitive, provide
  biased estimates of teacher probability distribution to the student, resulting
  in suboptimal performance and calibration. We propose an
  importance-sampling-based method `Random Sampling Knowledge Distillation',
  which provides unbiased estimates, preserves the gradient in expectation, and
  requires storing significantly sparser logits. Our method enables faster
  training of student models with marginal overhead (<10%) compared to
  cross-entropy based training, while maintaining competitive performance
  compared to full distillation, across a range of model sizes from 300M to 3B.
paper_authors._id: 67e4c860136c8a867191e52e
paper_authors.name: Anshumann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:03:45.930Z
paper_authors._id: 6627ff2a3b4fbc8420a416c3
paper_authors.avatarUrl: /avatars/de3aafdaf5563fe25edcdb92b394474f.svg
paper_authors.fullname: AR
paper_authors.type: user
paper_authors.user: Anshumann
paper_authors._id: 67e4c860136c8a867191e52f
paper_authors.name: Mohd Abbas Zaidi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:03:57.163Z
paper_authors._id: 61765fe0b0715831eab6d465
paper_authors.avatarUrl: /avatars/830406cebdd7ddf66799bf71f6bbff4b.svg
paper_authors.fullname: Mohd Abbas Zaidi
paper_authors.type: user
paper_authors.user: ya-mehdi
paper_authors._id: 67e4c860136c8a867191e530
paper_authors.name: Akhil Kedia
paper_authors._id: 67e4c860136c8a867191e531
paper_authors.name: Jinwoo Ahn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:04:07.687Z
paper_authors._id: 65d61eebee922bc7a777d5a6
paper_authors.avatarUrl: /avatars/180efd503ef412b4dc728e6aa477c16e.svg
paper_authors.fullname: Jinwoo Ahn
paper_authors.type: user
paper_authors.user: AndrewAhn
paper_authors._id: 67e4c860136c8a867191e532
paper_authors.name: Taehwak Kwon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:04:23.731Z
paper_authors._id: 629059cdb90dde28ef5cbb30
paper_authors.avatarUrl: /avatars/451b0e8999447b3a2f03378fe98c0661.svg
paper_authors.fullname: Taehwak Kwon
paper_authors.type: user
paper_authors.user: Rock222
paper_authors._id: 67e4c860136c8a867191e533
paper_authors.name: Kangwook Lee
paper_authors._id: 67e4c860136c8a867191e534
paper_authors.name: Haejun Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:04:39.155Z
paper_authors._id: 653b6e7232bd4db35d981615
paper_authors.avatarUrl: /avatars/0fbf0c5d502b840bf26baf8c420c7593.svg
paper_authors.fullname: Haejun Lee
paper_authors.type: user
paper_authors.user: haejunlee
paper_authors._id: 67e4c860136c8a867191e535
paper_authors.name: Joohyung Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:04:46.702Z
paper_authors._id: 64b4be0665a7e15eac085878
paper_authors.avatarUrl: /avatars/dd19ec2987fb0735457c6492b53aacfe.svg
paper_authors.fullname: Joo-hyung Lee
paper_authors.type: user
paper_authors.user: snrbs17
paper_title: Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
paper_summary: Knowledge distillation can be a cost-effective technique to distill knowledge
  in Large Language Models, if the teacher output logits can be pre-computed and
  cached. However, successfully applying this to pre-training remains largely
  unexplored. In this work, we prove that naive approaches for sparse knowledge
  distillation such as caching Top-K probabilities, while intuitive, provide
  biased estimates of teacher probability distribution to the student, resulting
  in suboptimal performance and calibration. We propose an
  importance-sampling-based method `Random Sampling Knowledge Distillation',
  which provides unbiased estimates, preserves the gradient in expectation, and
  requires storing significantly sparser logits. Our method enables faster
  training of student models with marginal overhead (<10%) compared to
  cross-entropy based training, while maintaining competitive performance
  compared to full distillation, across a range of model sizes from 300M to 3B.
paper_ai_summary: Random Sampling Knowledge Distillation provides unbiased and efficient knowledge transfer by sampling logits, enhancing training speed with minimal performance loss compared to full distillation.
paper_ai_keywords: sparse knowledge distillation knowledge distillation teacher output logits student model cross-entropy based training