Word embeddings learnt from large corpora have been adopted in various applications in natural language processing and served as the general input representations to learning systems. Recently, a series of post-processing methods have been proposed to boost the performance of word embeddings on similarity comparison and analogy retrieval tasks, and some have been adapted to compose sentence representations. The general hypothesis behind these methods is that by enforcing the embedding space to be more isotropic, the similarity between words can be better expressed. We view these methods as an approach to shrink the covariance/gram matrix, which is estimated by learning word vectors, towards a scaled identity matrix. By optimising an objective in the semi-Riemannian manifold with Centralised Kernel Alignment (CKA), we are able to search for the optimal shrinkage parameter, and provide a post-processing method to smooth the spectrum of learnt word vectors which yields improved performance on downstream tasks.
This paper presents a novel method for unsupervised post-processing of pretrained word embeddings that enforces the distributional word vector space to be more isotropic, which in turn improves the expressiveness and quality of the space in terms of similarity. The method is based on the shrinkage of the covariance/Gram matrix and its effects on the input space are evaluated across a range of intrinsic evaluation tasks. While I like the idea overall and this line of work in general, there are still some concerns with the current version of the paper:
* Overall, although its design seems more principled, the proposed method does not seem significantly better than the previous (very similar) method of Mu et al. I would like to see more evidence in the favour of the proposed method, pointing that we should use that instead of Mu's method. Also, the gains over non-processed spaces often seem insignificant, and offer only small benefits.
* The derivation of the method seems too verbose, especially in light of the fact that it is directly inspired by previous work on the shrinkage estimation of covariance matrices and CKA. I would suggest the authors to spend more time on linking their high-level hypotheses to the low-level mathematical implementations instead of flooding the paper with equations - for the interested reader a lot of the derivation process can be put into an appendix, the paper should focus on conveying the key principles instead. This would also offer additional space for more experiments.
* One very relevant paper is not mentioned at all: https://arxiv.org/pdf/1809.02094.pdf (Artetxe et al., CoNLL 2018). I would suggest the authors to cite that work and ideally even compare to it on their set of intrinsic tasks (e.g., word similarity, word analogy), and then discuss the difference in results and their approach to unsupervised post-processing. This shouldn't be so difficult to do as the code from that paper is available online: https://github.com/artetxem/uncovec
* The paper mixes true similarity datasets (such as SimLex) with broader semantic relatedness datasets (MEN, WordSim-353), even mixing true semantic similarity of the same dataset (WordSim353-SIM) with its relatedness subset (WordSim353-REL). In light of the known conceptual differences between the relations of similarity versus relatedness, I would suggest to report the results separately for the two tasks. For instance, another true similarity dataset, which is not used in the evaluation is SimVerb-3500. Along the same line, it is also not clear what type of similarity is meant when the authors state that through post-processing 'the similarity between words can be better expressed'. What does it mean to better express the similarity between words in the first place? Do we talk about true similarity or relatedness or both? However, the two relationships between words support different classes of downstream applications, so therefore it is even more problematic 1) not to distinguish between the two and 2) not to report any results in any downstream (extrinsic) tasks where the post-processed embeddings are used as features (STS is still a semi-intrinsic task imho; BLI is considered as an intrinsic task in cross-lingual settings).
* One of the key reasons to apply post-processing is to mitigate the frequency artefacts: however, such an evaluation that goes towards that direction is never executed. For instance, I would like to see a focused experiment that measures how post-processing affects high-frequency versus mid-to-lower frequency or rare words. A recently developed CARD-660 dataset might be used to this end.
* It is not clear how exactly the authors run Mu et al.'s method, that is, how many top principal components are removed for the input vectors? How is this selected? Are always the optimal results reported for the baseline Mu et al.'s method? Why not reporting the results with removing 2 and 3 at the same time to further prove the point that their method is non-automatic? This would also give the reader a hint how much the results with Mu's method actually differ/decrease if one just decides to make the method 'automatic' by just fixing the method to always remove the same number of top principal components.
Minor remarks:
* The title of the paper is a bit imprecise: in the word embedding literature, the term post-processing is often referred to the methods that fine-tune word embeddings using some external knowledge after (i.e., post) the initial distributional training (e.g., the so-called retrofitting methods). However, in the context of the paper post-processing actually refers to some unsupervised post-training steps on the input space without injecting any external information. This should be made clearer in the paper, and perhaps adding a paragraph which outlines the core difference to other work on retrofitting would be helpful as well.
* I might be missing something while reading Section 3, but it is currently not clear to me how the oracle Gram matrix K is obtained in the first place. Perhaps it makes sense to briefly summarize this in a quite direct way to avoid the reader's confusion?
* It is great to see a summary of the key post-processing steps at the very end of Section 3; this is really helpful for everyone who would like to try out the proposed method off-the-shelf. However, the summary is not self-contained as it is not clear what \mathcal{L}'' refers to (and the reader must search through the derivations again to find its meaning).
* I like the evaluation on word translation, and I believe that the proposed post-processing methods could actually improve word translation through some pre-alignment perturbations. It is a pity that the method is not evaluated on more distant language pairs, as I believe that the method might have much more effect there than on the already-saturated EN-to-ES/FR/IT bilingual lexicon induction tasks.
* For de-en word translation Mu's method actually beats the proposed method (incorrect number in bold)
* It is not clear why MUSE is used for word retrieval experiments, given the fact that it is known to be unstable (Sogaard et al., ACL 2019), and there are more robust and more effective methods available such as VecMap (Artetxe et al., ACL 2018) or RCSLS (Joulin et al., EMNLP 2019)