Neural embeddings have been used with great success in Natural Language Processing (NLP) where they provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks. The success of neural embeddings has prompted significant amounts of research into applications in domains other than language. One such domain is graph-structured data, where embeddings of vertices can be learned that encapsulate vertex similarity and improve performance on tasks including edge prediction and vertex labelling. For both NLP and graph-based tasks, embeddings in high-dimensional Euclidean spaces have been learned. However, recent work has shown that the appropriate isometric space for embedding complex networks is not the flat Euclidean space, but a negatively curved hyperbolic space. We present a new concept that exploits these recent insights and propose learning neural embeddings of graphs in hyperbolic space. We provide experimental evidence that hyperbolic embeddings significantly outperform Euclidean embeddings on vertex classification tasks for several real-world public datasets.
== Preamble ==
As promised, I have read the updated paper from scratch and this is my revised review. My original review is kept below for reference. My original review had rating "4: Ok but not good enough - rejection".
== Updated review ==
The revised improves upon the original submission in several ways and, in particular, does a much better job at positioning itself within the existing body of literature. The new experiments also indicate that the proposed model offer some improvement over Nickel & Douwe, NIPS 2017).
I do have remaining concerns that unfortunately still prevent me from recommending acceptance:
- Throughout the paper it is argued that we should embed into a hyperbolic space. Such a space is characterized by its metric, but the proposed model do not use a hyperbolic metric. Rather it relies on a heuristic similarity measure that is inspired by the hyperbolic metric. I understand that this may be a practical choice, but then I find it misleading that the paper repeatedly states that points are embedded into a hyperbolic space (which is incorrect). This concern was also raised on this forum prior to the revision.
- The resulting optimization is one of the key selling points of the proposed method as it is unconstrained while Nickel & Douwe resort to a constrained optimization. Clearly unconstrained optimization is to be preferred. However, it is not entirely correct (from what I understand), that the resulting optimization is indeed unconstrained. Nickel & Douwe work under the constraint that |x| < 1, while the proposed model use polar coordinates (r, theta): r in (0, infinity) and theta in (0, 2 pi]. Note that theta parametrize a circle, and therefore wrapping may occur (this should really be mentioned in the paper). The constraints on theta are quite easy to cope with, so I agree with the authors that they have a more simple optimization problem. However, this is only true since points are embedded on the unit disk (2D). Should you want to embed into higher dimensional spaces, then theta need to be confined to live on the unit sphere, i.e. |theta| = 1 (the current setting is just a special-case of the unit sphere). While optimizing over the unit sphere is manageable it is most definitely a constrained optimization problem, and it is far from clear that it is much easier than working under the Nickel & Douwe constraint, |x| < 1.
Other comments:
- The sentence "even infinite trees have nearly isometric embeddings in hyperbolic space (Gromov, 2007)" sounds cool (I mean, we all want to cite Gromov), but what does it really mean? An isometric embedding is merely one that preserves a metric, so this statement only makes sense if the space of infinite trees had a single meaningful metric in the first place (it doesn't; that's a design choice).
- In the "Contribution" and "Conclusion" sections it is claimed that the paper "introduce the new concept of neural embeddings in hyperbolic space". I thought that was what Nickel & Douwe did... I understand that the authors are frustrated by this parallel work, but at this stage, I don't think the present paper can make this "introducing" claim.
- The caption in Figure 2 miss some indication that "a" and "b" refer to subfigures. I recommend "a" --> "a)" and "b" --> "b)".
- On page 4 it is mentioned that under the heuristic similarity measure some properties of hyperbolic spaces are lost while other are retained. From what I can read, it is only claimed that key properties are kept; a more formal argument (even if trivial) would have been helpful.
== Original Review ==
The paper considers embeddings of graph-structured data onto the hyperbolic Poincare ball. Focus is on word2vec style models but with hyperbolic embeddings. I am unable to determine how suitable an embedding space the Poincare ball really is, since I am not familiar enough with the type of data studied in the paper. I have a few minor comments/questions to the work, but my main concern is a seeming lack of novelty:
The paper argues that the main contribution is that this is the first neural embedding onto a hyperbolic space. From what I can see, the paper
  Poincaré Embeddings for Learning Hierarchical Representations
  https://arxiv.org/abs/1705.08039
consider an almost identical model to the one proposed here with an almost identical motivation and application set. Some technicalities appear different, but (to me) it seems like the main claimed novelties of the present paper has already been out for a while. If this analysis is incorrect, then I encourage the authors to provide very explicit arguments for this in the rebuttal phase.
Other comments:
*) It seems to me that, by construction, most data will be pushed towards the boundary of the Poincare ball during the embedding. Is that a property you want?
*) I found it rather surprising that the log-likelihood under consideration was pushed to an appendix of the paper, while its various derivatives are part of the main text. Given the not-so-tight page limits of ICLR, I'd recommend to provide the log-likelihood as part of the main text (it's rather difficult to evaluate the correctness of a derivative when its base function is not stated).
*) In the introduction must energy is used on the importance of large data sets, but it appears that only fairly small-scale experiments are considered. I'd recommend a better synchronization.
*) I find visual comparisons difficult on the Poincare ball as I am so trained at assuming Euclidean distances when making visual comparisons (I suspect most readers are as well). I think one needs to be very careful when making visual comparisons under non-trivial metrics.
*) In the final experiment, a logistic regressor is fitted post hoc to the embedded points. Why not directly optimize a hyperbolic classifier?
Pros:
+ well-written and (fairly) well-motivated.
Cons:
- It appears that novelty is very limited as highly similar work (see above) has been out for a while.