Llm, Knowledge Distillation, Quantization, Online Advertising, AI.
on reducing combined cross-entropy loss between student output and groundtruth + KL divergence between prob. distribution of student and teacher for same input. However, the former method is preferred for its simplicity and generic framework that is applicable to multiple teacher models. Model