Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, Distillation.
we want to learn the student embedding s(x) with parameters θs that mimics the same behavior as t(x) if used for a downstream supervised task e.g., image classification. 1.2. Framework For simplicity, ti=t(xi) for the embedding of the model t(x) on the input image xi normalized by l2 norm. We have