Dynamic Temperature Knowledge Distillation
<|reference_start|>Dynamic Temperature Knowledge Distillation: Temperature plays a pivotal role in moderating label softness in the realm of knowledge distillation (KD). Traditional approaches often employ a static temperature throughout the KD process, which fails to address the nuanced complexities of samples with varying levels of difficulty and overlooks the distinct capabilities of different teacher-student pairings. This leads to a less-than-ideal transfer of knowledge. To improve the process of knowledge propagation, we proposed Dynamic Temperature Knowledge Distillation (DTKD) which introduces a dynamic, cooperative temperature control for both teacher and student models simultaneously within each training iterafion. In particular, we proposed "\textbf{sharpness}" as a metric to quantify the smoothness of a model's output distribution. By minimizing the sharpness difference between the teacher and the student, we can derive sample-specific temperatures for them respectively. Extensive experiments on CIFAR-100 and ImageNet-2012 demonstrate that DTKD performs comparably to leading KD techniques, with added robustness in Target Class KD and None-target Class KD scenarios.The code is available at https://github.com/JinYu1998/DTKD.<|reference_end|>
@article{wei2024dynamic,
    title={Dynamic Temperature Knowledge Distillation},
    author={Yukang Wei, Yu Bai},
    journal={arXiv preprint arXiv:2404.12711},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.12711},
    primaryClass={cs.LG cs.AI cs.CV}
}