[cls] token. When it is not explicitly needed by the method, using the average of
the patch tokens instead of the class token saves memory without much change on the
accuracies of the network [Zhai et al., 2022a].

3.6 Techniques for High Performance Masked Image Modeling
While there are several approaches to masked pretraining, the state-of-the-art systems
that employ them tend to pair MIM with other techniques. For example, the ConvNextV2
architecture, which was state of the art on ImageNet (for models trained with only public
data) when released, employs MAE pretraining [Woo et al., 2023]. Interestingly, the authors
point out that simply pretraining a ConvNextV2 with the MAE framework is subpar. They
propose adding a novel normalization layer, called “global response normalization,” that
proves vital to reaching state-of-the-art results [Woo et al., 2023].
A Cookbook of Self-Supervised Learning