The second reason is linked to the architecture. Since transformers [Vaswani et al.,
2017] are employed in almost all domains of computer science, many works aim to reduce
the compute and memory requirements of the attention mechanism. One approach is with
low-rank and/or sparse approximation mechanisms [Kitaev et al., 2020, Choromanski
et al., 2020, Wang et al., 2020a, Chen et al., 2021a, Zaheer et al., 2020]. For instance, Li
et al. [2022b] use sparse self-attention to improve eﬃciency in the context of SSL vision

4https://github.com/vturrisi/solo-learn
5https://github.com/facebookresearch/FFCV-SSL

36
A Cookbook of Self-Supervised Learning