These speed-ups are available in open-source libraries: Fairseq [Ott et al., 2019],
FairScale [FairScale, 2021], XFormers [Lefaudeux et al., 2022], Apex [NVidia, 2021], etc.
Another simple way to speed up the training of vision transformers is to use Pytorch
bﬂoat16 which allow faster training while keeping the same precision range as ﬂoat32
(this is useful to avoid the usual numerical instability issues one can encounter when
training vision transformers in ﬂoat16).

4 Extending Self-Supervised Learning Beyond Images and

Classiﬁcation
A Cookbook of Self-Supervised Learning