2.6.2 Dimensional Collapse of Representations
While the goal of joint self-supervised methods is to learn meaningful representations,
a signiﬁcant part of the approaches suﬀer from what is called dimensional collapse.
Dimensional collapse occurs when information encoded across diﬀerent dimensions of the
representation is redundant. In other words in the output of the projector, the embeddings
are rank-deﬁcient, which can be approximated via the singular value spectrum of the
embeddings, as illustrated in Figure 12.
A Cookbook of Self-Supervised Learning