Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, NLP.
Targets Training targets are constructed based on the output of the top-K blocks of the teacher network for time-steps which are masked in student-mode. The output of block l at time-step t is denoted as alt. A normalization is applied to each block to obtain ˆalt before averaging the top K blocks