POINCARÉ GLOVE: HYPERBOLIC WORD EMBEDDINGS
Words are not created equal. In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal. In this paper, justified by the notion of delta-hyperbolicity or tree-likeliness of a space, we propose to embed words in a Cartesian product of hyperbolic spaces which we theoretically connect to the Gaussian word embeddings and their Fisher geometry. This connection allows us to introduce a novel principled hypernymy score for word embeddings. Moreover, we adapt the well-known Glove algorithm to learn unsupervised word embeddings in this type of Riemannian manifolds. We further explain how to solve the analogy task using the Riemannian parallel transport that generalizes vector arithmetics to this new type of geometry. Empirically, based on extensive experiments, we prove that our embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection. In particular, for word hypernymy, we obtain new state-of-the-art on fully unsupervised WBLESS classification accuracy.
The English, grammar and writing style is very good, as are the citations.
The technical quality appears to me to be very good (I am not an expert in Poincare spaces).
The authors demonstrate a good knowledge of the mathematical theory with the constructions made in Section 6.
The experimental write-up has been abbreviated.  The lexical entailment results Tables 6 and 7 are just sitting there without discussion, as far as I can see, as are the qualitative results Tables 4 and 5.   The entailment results are quite complex and really need supporting interpretation.  For instance, for Hyperlex, WN-Poincare is 0.512, above yours.
For your entailment score you say "For simplicity, we propose dropping the dependence in μ".  This needs more justification and discussion as it is counter-intuitive for those not expert in  Poincare spaces.
Section 6.2 presents the entailment score.  Note Nickel etal. give us a nice single formula.  You however, provide 4 paragraphs of construction from which an astute reader would then have to work on to extract your actual method.  I would prefer to see a summary algorithm given somewhere.  Perhaps you need another appendix.
RADAGRAD is discussed in Section 5, but I'd have preferred to see it discussed again in Section 8 and discussed to highlight what was indded done and the differences.  It certainly makes the paper non-reproducible.
A significant part of the theory in earlier sections is about the 50x2D method, but in experiments this doesn't seem to work as well.  Can you justify this some other how:  its much faster, its more interpretable?  Otherwise, I'm left thinking, why not delete this stuff?
The paper justifies its method with a substantial and winning comparison against vanilla GloVe.  That by itself is a substantial contribution.
But now, one is then hit with a raft of questions.  Embedding methods are popping up like daisies all over the fields of academia.  Indeed, word similarity and lexical entailment tasks themselves are proliferating too.  To me, its really unclear what one needs to achieve in the empirical section of a paper.  To make it worse, some folks use 500D, some 100D, some 50D, so results aren't always comparible.  Demonstrating one's work is state-of-the-art against all comers is a massive implementation effort.  I notice some papers now just compare against one other (e.g., Klami etal. ECML-PKDD, 2018).
My overall feeling is that this paper tries to compress too much into a small space (8 pages).
I think it really needs to be longer to present what is shown.   Moreover, I would want to see the inclusion of the work on 50x2D justified. So my criticisms are about the way the paper is written, not about the quality of the work.  
Moroever, though, one needs to consider comparisons against models other than GloVe.
Addendum:  You know, what I really love about ICLR is the effort authors make to refresh their paper and respond to reviewers.  You guys did a great job.  Really impressed.  50x2D now clarified and some of the hasty/unexplained bits fixed.