prediction [Radford et al., 2018, 2019, Brown et al., 2020] is akin to masking the last token
in a string, while bidirectional encoders mask tokens anywhere in the string [Devlin et al.,
2018] or ﬁll larger spans of missing text [Raﬀel et al., 2020, Tay et al., 2022]. This choice of
unidirectional next-token prediction versus bidirectional approaches leads to meaningful
diﬀerences in downstream text applications [Artetxe et al., 2022]. For contrastive learning,
positive pairs often come from masking and/or cropping input sequences [Meng et al.,
2021, Giorgi et al., 2021]. They can also be generated using dropout so that one input
has two diﬀerent latent representations [Gao et al., 2021]. Additionally, some methods
for both contrastive and reconstructive pretraining corrupt the input with several other
augmentations including document rotation, sentence permutation, and token deletion
[Lewis et al., 2020, Raﬀel et al., 2020, Wu et al., 2018].
A Cookbook of Self-Supervised Learning