256512102420488192Embedding dimension6062646668Online top-1 accuracyVICReg256512102420488192Embedding dimension6062646668VICReg-exp256512102420488192Embedding dimension6062646668VICReg-ctr256512102420488192Embedding dimension6062646668SimCLR-Orig.256512102420488192Embedding dimension6062646668SimCLR-Tunedd-d-d8192-8192-d2048-d2048-2048-drepresentations lead to better linear probe performance when using CISSL. Bordes et al.
[2023b] investigated more deeply the impact of the backbone dimension across common
SSL methods like VICReg, SimCLR or BYOL. They show that traditional supervised methods
decline in performance when the dimension of the backbone is increased. On the other
hand, SSL methods highly beneﬁt from wider backbone representations as shown in
Figure 12a.
In fact, it is much more beneﬁcial in SSL to increase the backbone size
when training a ResNet than increasing the width or depth of the ResNet as illustrated
A Cookbook of Self-Supervised Learning