Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, Distillation.
where τ is the temperature. The loss for a particular query point is estimated as the KL divergence between the probabilities over all anchor points under the teacher and student models, and this loss is summed over all query points: where pi(s) is the probability distribution of query i over all