Lφ,ψ(θ) =

min

θ

ψ((cid:107)zi − zi(cid:48)(cid:107)2

2 − (cid:107)zi − zj(cid:107)2
2)

(15)

where z are representations with indices i and j running from 1 to N. With diﬀerent φ and
ψ, Eqn. 15 covers many loss functions (Figure 8). In particular, setting φ(x) = τ log((cid:15) + x)

 .

(cid:88)

j(cid:54)=i

N(cid:88)

i=1

φ

17

𝒙[𝑖]𝒙[𝑖′]𝒛[𝑖]𝒛[𝑖′]𝒙[𝑗]𝒛[𝑗]𝑑!"𝑑!#"ℒ!,#Multi-layer network 𝜽and ψ(x) = exp(x/τ ) gives a generalized version of InfoNCE loss [Oord et al., 2018]:

Lnce :=−τ

N(cid:88)

i=1

e−(cid:107)zi−zi(cid:48)(cid:107)2

2/τ +(cid:80)

2/τ

log

(cid:15)e−(cid:107)zi−zi(cid:48)(cid:107)2

j(cid:54)=i e−(cid:107)zi−zj(cid:107)2

2/τ

(16)

where (cid:15) > 0 is some constant e.g. (cid:15) = 1 has been used in He et al. [2020b], Tian et al.
[2020a], (cid:15) = 0 yields a slight variation of SimCLR [Chen et al., 2020b], the DCL loss [Yeh
et al., 2021].
A Cookbook of Self-Supervised Learning