Another simple way to evaluate SSL methods, called RankMe, was introduced by Garrido
et al. [2022a]. The idea is to use the eﬀective rank of representations, deﬁned as the

32

entropy of the singular value distribution of the embeddings. It can be computed as:

−

min(N,K)(cid:88)

k=1

 , pk =

RankMe(Z) = exp

pk log pk

σk(Z)
(cid:107)σ(Z)(cid:107)1

+ (cid:15)

(17)

It is shown to be a necessary condition for good performance, though you can achieve full
rank representations with degenerate results (e.g. a random matrix with entries sampled
i.i.d.
from a Gaussian distribution). While this cannot be used to evaluate diﬀerent
methods, it works well for hyperparameter selection, as shown in Table 3.
A Cookbook of Self-Supervised Learning