  Prior work investigating the geometry of pre-trained word embeddings have
shown that word embeddings to be distributed in a narrow cone and by centering
and projecting using principal component vectors one can increase the accuracy
of a given set of pre-trained word embeddings. However, theoretically, this
post-processing step is equivalent to applying a linear autoencoder to minimise
the squared l2 reconstruction error. This result contradicts prior work (Mu and
Viswanath, 2018) that proposed to remove the top principal components from
pre-trained embeddings. We experimentally verify our theoretical claims and
show that retaining the top principal components is indeed useful for improving
pre-trained word embeddings, without requiring access to additional linguistic
resources or labelled data.

Computation and Language