distribution to anchors from the student distribution using KL-divergence that relaxes the
binary distinction between positive and negative samples. MSF [Koohpayegani et al., 2021]
compares a query’s nearest neighbor representation to the student target’s representation
and then minimize the (cid:96)2 distnace between them with renormalization (akin to cosine
similarity maximization). Another approach, SSCD builds on the contrastive objective
to the task of copy detection outperforming copy detection models and other contrastive
methods [Pizzi et al., 2022]. Aside from the widespread use of the contrastive objective,
many more methods employ similar running average updates as part of their training
mechanism. For example, self-distillation [Hinton et al., 2015, Furlanello et al., 2018],
Deep Q Network in reinforcement learning [Mnih et al., 2013], Mean Teacher in semi-
supervised learning [Tarvainen and Valpola, 2017], and even model average in supervised
A Cookbook of Self-Supervised Learning