3.5.2 Role of Learning Rate (Schedulers) and Optimizers
Here we overview typical standard settings for learning rate schedulers and optimizers
across methods. To determine the learning rate, methods often scale a base learning rate
based on the batch size according to the heuristic by Goyal et al. [2017]: learning rate
∗ base learning rate. For ImageNet pretraining, VICREg, Barlow Twins, BYOL,
= batch size
and SimCLR use a base learning rate of 0.2 − 0.3 with the LARS optimizer [You et al.,
2017]. Additionally for some methods such as Barlow twins, a much smaller learning rate
(0.0048) is used to update the bias terms and batch norm parameters. Other methods
such as MAE, DINO, and iBot use the AdamW optimizer [Loshchilov and Hutter, 2017]
with a smaller base learning rate of 1e − 5 − 5e − 4. For a discussion of weight decay see
Section 3.5.3. The most common training schedule involves a warmup period, usually 10
A Cookbook of Self-Supervised Learning