results). It has been shown that contrasting methods show inferior performance than
masked image modeling with regards to ﬁne-tuning, because they are less “optimization
friendly” [Wei et al., 2022] - which explains the overall interest over MIM. It is by far the
most computationally expensive of the evaluation methods, since it needs to re-train the
whole network. The most common benchmark on ImageNet runs the optimization over
100 epochs for ViT smaller than base, and for 50 epochs for larger models [He et al., 2022].
Other works [Bao et al., 2021b, Peng et al., 2022, Wang et al., 2022b] ﬁrst ﬁne-tune on
ImageNet-21k for 60 epochs, and further ﬁne-tune on ImageNet-1k, which represents
between 1/5 to 2 times the cost of the pre-training phase.
A Cookbook of Self-Supervised Learning