  Knowledge distillation is the process of transferring the knowledge from a
large model to a small model. In this process, the small model learns the
generalization ability of the large model and retains the performance close to
that of the large model. Knowledge distillation provides a training means to
migrate the knowledge of models, facilitating model deployment and speeding up
inference. However, previous distillation methods require pre-trained teacher
models, which still bring computational and storage overheads. In this paper, a
novel general training framework called Self Distillation (SD) is proposed. We
demonstrate the effectiveness of our method by enumerating its performance
improvements in diverse tasks and benchmark datasets.

Machine Learning;Computer Vision and Pattern Recognition