The distillation of domain-speciﬁc knowledge from LLMs into more compact neural
networks has emerged as a promising area. Considering the specialty of LLMs, recent
techniques of Knowledge Distillation can be divided into two streams: 1) White-box
Knowledge Distillation: the teacher model’s parameters are available to use; 2) Black-
box Knowledge Distillation: only the teacher model’s predictions are accessible.
White-box knowledge distillation. This approach not only aims to substantially
decrease inference latency but also to amplify the eﬀectiveness of specialized task-
solving capabilities. A compelling example is the work by Muhamed et al., who
ingeniously compressed a behemoth 1.5 billion-parameter white-box LLM into a far
more manageable 70 million-parameter model. This was speciﬁcally engineered for
optimizing Click-Through Rate (CTR) prediction tasks. They introduced an innova-
tive architecture featuring twin-structured BERT-like encoders coupled with a fusion
Beyond Efficiency