LayerDecay [Clark et al., 2020] decreases the learning rate geometrically along the
layers. Put diﬀerently, the last layer is not aﬀected, while the ﬁrst has very small learning
rate.
In SSL vision models, LayerDecay increases performance when ﬁne-tuning on

3https://generallyintelligent.ai/blog/2020-08-24-understanding-self-supervised-contrastive-learning/

28

downstream tasks [Bao et al., 2021b, Zhou et al., 2022a, He et al., 2022]. Depending
on the model size, the parameter is set between 0.65 to 0.85 – larger models usually need
higher values because there are more layers. The underlying principle is that SSL builds
strong model backbones, therefore we only need to ﬁne-tune the shallowest layers.

LayerScale [Touvron et al., 2021b] is a per-channel multiplication of the vector produced
by each residual block of the transformer. It increases the stability of the optimization
and permits deeper ViT (larger than ViT-B).
A Cookbook of Self-Supervised Learning