instabilities). This leads to approximately 50% reduction in communication costs compared to the ﬂoat32
gradient all-reduce operation used in DistributedDataParallel (DDP), which is used in other self-supervised
pretraining methods (Caron et al., 2021; Zhou et al., 2021). As a consequence, the training procedure
scales more eﬃciently than DDP with ﬂoat16 autocast when scaling the number of GPU nodes. Overall,
Pytorch-FSDP mixed-precision is superior to DDP with autocast in virtually all cases we encountered.
DINOv2- Learning Robust Visual Features without Supervision