not achieve competitive performance with supervised learning on downstream tasks, and
pre-dates vision transformer architectures which modern masked training routines build
upon. Subsequently, BERT [Devlin et al., 2019] shook up the natural language processing
world by replacing text tokens input to a transformer language model with learnable
mask tokens and teaching the model to recover the original text. This paradigm, termed
masked language modeling (MLM), can also be interpreted as a form of the above strategy,
degrading a sample via masking and teaching a model to undo the masking degradation.
MLM, along with span-inﬁlling techniques, remains popular as a SSL objective for large
language models [Raﬀel et al., 2020, Wang et al., 2022a, Tay et al., 2022].
A Cookbook of Self-Supervised Learning