Deep Learning, Artificial Intelligence, Self Supervised Learning, Image Classification, Unsupervised Learning.
can get zb and zk from the base model and the k-th expert model respectively. The Distillation objective is then the average of the two mean squared errors: Noted that the outputs of rb and rk are not ℓ2 normalized. 2.2. Datasets Datasets The first dataset is a private dataset of roughly 300