Aside from forming positives using semantic preserving transformations, mining pos-
itive pairs naturally arising in data is also possible. An iconic triplet loss is based on
video frames where the positive pairs come from nearby frames (while negatives are from
far away frames) developed in Sermanet et al. [2018] coined Time-Contrastive (TC) Time-
Contrastive Learning. Nonlinear ICA [Hyvarinen and Morioka, 2016] introduced a proof
that you can learn the log PDF when doing classiﬁcation tasks. Alexey et al. [2015] trains
a classiﬁcation pretext task by transforming image patches in comparison to diﬀerent
transformations of image patches. One disadvantage is that this setup can involve too
many classes leading performance to degrade on downstream tasks. To overcome this,
NCE has been successfully employed in Mnih and Teh [2012], Mnih and Kavukcuoglu
[2013] to modify the denominator in order not to loop over all classes. This is an alternative
A Cookbook of Self-Supervised Learning