Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, Distillation.
Framework 1.1. Goal The goal is to train a deep model (e.g. ResNet-50) using an off-the-shelf self-supervised learning algorithm and then, compress it to a less deep model (e.g., AlexNet). Assuming a frozen teacher embedding t(x) with parameters θt that maps an image x into an N-D feature space,