3.2 Role of the Projector
Most SSL with joint embedding methods include a projector (usually 2- or 3-layers MLP
with ReLU) after the encoder. The SSL loss is applied to the projector’s output, and the
projector is usually discarded after training. This crucial component was introduced in
SimCLR [Chen et al., 2020b] and, although not responsible for avoiding collapse, allows
signiﬁcant top-1 accuracy gains on ImageNet. For example, in a 100-epochs training, the
projector adds around 20% of top-1 accuracy in SimCLR and VICReg (from around 50% to
68% and 48% to 68% respectively).
A Cookbook of Self-Supervised Learning