Another key ingredient along with the InfoNCE loss used in SimCLR is the non-
parametric softmax introduced by Wu et al. [2018]. This name is motivated by removing
the need to have a "parametrized" linear layer on top of the representation to compute
the softmax by instead comparing representations with each others. This loss formu-
lation already contained a temperature parameter in the softmax which is responsible
for increasing or decreasing the sharpness of events in predictions. Other noteworthy
developements include Schroﬀ et al. [2015] use triplet loss with active triplet selection
(hard positive, hard negative) either online from the current mini-batch or from a past
checkpoint akin to momentum networks (discussed in section 2.3). Weinberger and Saul
[2009] introduced push-pull weighting, to push negatives apart while pulling positives
together, in a triplet loss to increase the margin of K-NN based models. Tian et al. [2020a]
introduced the possibility of many positive views.
A Cookbook of Self-Supervised Learning