Distributional Semantics Models(DSM) derive word space from linguistic items in context. Meaning is obtained by defining a distance measure between vectors corresponding to lexical entities. Such vectors present several problems. This work concentrates on quality of word embeddings, improvement of word embedding vectors, applicability of a novel similarity metric used 'on top' of the word embeddings. In this paper we provide comparison between two methods for post process improvements to the baseline DSM vectors. The counter-fitting method which enforces antonymy and synonymy constraints into the Paragram vector space representations recently showed improvement in the vectors' capability for judging semantic similarity. The second method is our novel RESM method applied to GloVe baseline vectors. By applying the hubness reduction method, implementing relational knowledge into the model by retrofitting synonyms and providing a new ranking similarity definition RESM that gives maximum weight to the top vector component values we equal the results for the ESL and TOEFL sets in comparison with our calculations using the Paragram and Paragram + Counter-fitting methods. For SIMLEX-999 gold standard since we cannot use the RESM the results using GloVe and PPDB are significantly worse compared to Paragram. Apparently, counter-fitting corrects hubness. The Paragram or our cosine retrofitting method are state-of-the-art results for the SIMLEX-999 gold standard. They are 0.2 better for SIMLEX-999 than word2vec with sense de-conflation (that was announced to be state-of the-art method for less reliable gold standards). Apparently relational knowledge and counter-fitting is more important for judging semantic similarity than sense determination for words. It is to be mentioned, though that Paragram hyperparameters are fitted to SIMLEX-999 results. The lesson is that many corrections to word embeddings are necessary and methods with more parameters and hyperparameters perform better.
This paper proposes a ranking-based similarity metric for distributional semantic models. The main idea is to learn "baseline" word embeddings, retrofitting those and applying localized centering, to then calculate similarity using a measure called "Ranking-based Exponential Similarity Measure" (RESM), which is based on the recently proposed APSyn measure.
I think the work has several important issues:
1. The work is very light on references. There is a lot of previous work on evaluating similarity in word embeddings (e.g. Hill et al, a lot of the papers in RepEval workshops, etc.); specialization for similarity of word embeddings (e.g. Kiela et al., Mrksic et al., and many others); multi-sense embeddings (e.g. from Navigli's group); and the hubness problem (e.g. Dinu et al.). For the localized centering approach, Hara et al.'s introduced that method. None of this work is cited, which I find inexcusable. 
2. The evaluation is limited, in that the standard evaluations (e.g. SimLex would be a good one to add, as well as many others, please refer to the literature) are not used and there is no comparison to previous work. The results are also presented in a confusing way, with the current state of the art results separate from the main results of the paper. It is unclear what exactly helps, in which case, and why. 
3. There are technical issues with what is presented, with some seemingly factual errors. For example, "In this case we could apply the inversion, however it is much more convinient [sic] to take the negative of distance. Number 1 in the equation stands for the normalizing, hence the similarity is defined as follows" - the 1 does not stand for normalizing, that is the way to invert the cosine distance (put differently, cosine distance is 1-cosine similarity, which is a metric in Euclidean space due to the properties of the dot product). Another example, "are obtained using the GloVe vector, not using PPMI" - there are close relationships between what GloVe learns and PPMI, which the authors seem unaware of (see e.g. the GloVe paper and Omer Levy's work). 
4. Then there is the additional question, why should we care? The paper does not really motivate why it is important to score well on these tests: these kinds of tests are often used as ways to measure the quality of word embeddings, but in this case the main contribution is the similarity metric used *on top* of the word embeddings. In other words, what is supposed to be the take-away, and why should we care?
As such, I do not recommend it for acceptance - it needs significant work before it can be accepted at a conference.
Minor points:
- Typo in Eq 10
- Typo on page 6 (/cite instead of \cite)