This phenomenon was ﬁrst illustrated by Hua et al. [2021] where the use of a whitening
batch normalization helped alleviate collapse. Dimensional collapse was also studied
from a theoretical point of view by Jing et al. [2022] with a focus on contrastive methods.
Several following works linked dimensional collapse to an impact on performance [He
and Ozay, 2022, Ghosh et al., 2022, Li et al., 2022a, Garrido et al., 2022a]. Some works
focused on unsupervised evaluation [Ghosh et al., 2022, Garrido et al., 2022a] where
dimensional collapse was found to be a good proxy for downstream performance.
Diﬀerent measures of dimensional collapse have been introduced such as the entropy of

18

Figure 9: Illustration of dimensional collapse before the projector (Left), and after the
projector (Right). Methods suﬀer from diﬀerent levels of collapse after the projector; while
no such collapse occurs for representations before the projector.
A Cookbook of Self-Supervised Learning