Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, Distillation.
do not see any reduction in accuracy by removing the momentum. Caching Teacher: Caching reduces the accuracy by only a small margin 53.4% to 53.0% on NN and 61.7% to 61.2% on linear evaluation while reducing the running time by a factor of almost 3. For experiments using ResNet-50×4, it cannot