Hyperbolic Neural Networks
Hyperbolic spaces have recently gained momentum in the context of machine learning due to their high capacity and tree-likeliness properties. However, the representational power of hyperbolic geometry is not yet on par with Euclidean geometry, mostly because of the absence of corresponding hyperbolic neural network layers. This makes it hard to use hyperbolic embeddings in downstream tasks. Here, we bridge this gap in a principled manner by combining the formalism of Möbius gyrovector spaces with the Riemannian geometry of the Poincaré model of hyperbolic spaces. As a result, we derive hyperbolic versions of important deep learning tools: multinomial logistic regression, feed-forward and recurrent neural networks such as gated recurrent units. This allows to embed sequential data and perform classification in the hyperbolic space. Empirically, we show that, even if hyperbolic optimization tools are limited, hyperbolic sentence embeddings either outperform or are on par with their Euclidean variants on textual entailment and noisy-prefix recognition tasks.
A number of recent works in deep learning have been using Hyperbolic Embeddings to represent features in Neural Networks. One main advantage of these representations is that they use a more natural geometry to represent hierarchical data such as trees. This work seeks a generic theoretical grounding as well as practical technique for the use of Hyperbolic Geometry at various layers of a neural network.  Pros: * Overall well written and clear; * A lot of the ideas are well motivated, and the authors took care to give intuitive justifications for many of their choices; * Hyperbolic geometry has been used in multiple machine learning papers recently, and this new paper is a good attempt at proposing a general framework; * I found the use of parallel transport to define biases in hyperbolic networks, particularly nice and clever.  Cons: * experiments are very limited in details (in the main part of the paper), since this is mostly a theory paper, I see this as a minor point; * the introduction points to a number of recent works using hyperbolic geometry. It would be good to have a bit more details about differences between these approaches, and the one proposed in the paper.  Minor/easily fixed comments: * A number of things about the definition of a geodesic on lines 70-71 are not quite right. Firstly, there are many paths of minimal lengths between points on a Riemannian manifold that are not geodesics (in the Riemannian sense). Indeed, the geodesic paths are only the ones that are parameterised proportionally to arc length. For ex, on the real line R, the path $\gamma(t) = t^2$ is of minimal length between $\gamma(0)$ and $\gamma(1)$, but it is not a geodesic. Secondly, on a general Riemannian manifold, there can be paths between two points that are not of minimal length, but are geodesics nonetheless. For example, on a sphere, for two points close to each other, there are two geodesics along the great circle through these two points. What is true, is that geodesics are locally of minimal length. On the Poincaré ball, there is always only one geodesic that joins any two points. The author should make it clear that they use this very particular property, and that it is not clear in general. * On line 72, parallel transport is introduced as a transformation that depends only on the starting and end point $P_{x\rightarrowy}$, but parallel transport is a transformation that does depend on the whole path. In this paper, the authors implicitly always choose the unique geodesic between two points on the Poincaré ball. This should be made clear. * Definition of exponential map on line 76 is unclear: the paragraph makes it look like it is defined only for tangent vectors of norm 1. Which is incorrect. * On line 106, the sum of x and 0 should be x, not 0. * Around line 113, I would suggest adding that $(-1)\bigotimes_c x = -x$. * In Theorem 4, here as well the authors should make it clear that this is for parallel transport along a geodesic. * In the 'Reformulating Euclidean MLR' section, from line 177, the first proposed parameterisation of hyperplanes as $H_{a,b}$ is not actually used. Only the second one (which is a very natural parameterisation), $\tilde{H}_{a, p}$, is used. Introducing both parameterisations is confusing. Removing the first one would also save space.  More important comments: * All hyperbolic balls of the same dimension are isomorphic. Indeed the map $x \mapsto \frac{\sqrt{c}}{\sqrt{c'}} x$ is an isometric diffeomorphism that transforms $\bigoplus_c$ into $\bigoplus_{c'}$, and so on. This is an important point. And using this point would help declutter many of the formulas: the $c$ subscript could be dropped in most places, since the equivalent formula for a general $c$ can easily be deduced from the case $c=1$. The only time that $c$ is useful in the paper is when the authors consider what happens when $c$ goes to $0$. Note also that in the experiments, only $c=1$ is actually used. * Regarding the section on hyperparameters, at line 552, could the authors clarify what they mean by "the hyperbolic geometry is more pronounced at the ball border"? The geometry is constant across the whole Poincaré ball. In particular, all points are equal and for any point, there exists an isometry of the ball that brings this point to 0. The only particular thing for the border is due to the particular parameterisation of the Poincaré ball: as we approach the boundary (in the sense of Euclidean metric, since in terms of hyperbolic metric, the border is always infinitely far), we lose precision in our compute.  =====================  Thanks to the author for their replies. I have read them, and indeed think this is very good submission. Please, do point out the isomorphisms between the balls with various 'c'. This is the main confusing point in the paper.