Fully-Sharded Data Parallel (FSDP). Minimizing our objective with the AdamW optimizer requires
4 model replicas in ﬂoat32 precision – student, teacher, optimizer ﬁrst moments, optimizer second moments.
This sums to 16 GB of memory for a billion-parameter model such as our ViT-g. In order to reduce this
memory footprint per GPU, we split the model replicas across GPUs, i.e., sharding 16 GB across GPUs
using the PyTorch implementation of FSDP. Consequently, the model size is not bounded by the memory of
a single GPU but by the total sum of GPU memory across compute nodes. The Pytorch implementation of
FSDP brings a second advantage, which is to save on the cross-GPU communication costs: the weight shards
are stored in ﬂoat32 precision as required by the optimizer, but broadcasting weights and reducing gradients
is done in ﬂoat16 precision for the backbone (MLP heads gradients are reduced in ﬂoat32 to avoid training
DINOv2- Learning Robust Visual Features without Supervision