Scaling self-supervised pretraining. A growing body of work has focused on the scaling abilities of
self-supervised learning in terms of data and model size (Caron et al., 2019; Goyal et al., 2019; Tian et al.,
2021; Goyal et al., 2022a). Most of these works use large quantities of uncurated data to train models
without supervision. They show evidence that discriminative methods scale with data, but because of the
poor quality of the pretraining data, most of the results are obtained by ﬁnetuning the features. Of particular
interest, Goyal et al. (2021) have also shown that these methods beneﬁt from scaling in model size given
enough pretrained data. This line of work questions the ability of self-supervised methods to work on any
data while we focus on producing the best pretrained encoders.
DINOv2- Learning Robust Visual Features without Supervision