HYPERBOLIC ATTENTION NETWORKS
Recent approaches have successfully demonstrated the benefits of learning the parameters of shallow networks in hyperbolic space. We extend this line of work by imposing hyperbolic geometry on the embeddings used to compute the ubiquitous attention mechanisms for different neural networks architectures. By only changing the geometry of embedding of object representations, we can use the embedding space more efficiently without increasing the number of parameters of the model. Mainly as the number of objects grows exponentially for any semantic distance from the query, hyperbolic geometry -as opposed to Euclidean geometry-can encode those objects without having any interference. Our method shows improvements in generalization on neural machine translation on WMT'14 (English to German), learning on graphs (both on synthetic and real-world graph tasks) and visual question answering (CLEVR) tasks while keeping the neural representations compact.
This paper replaces the dot-product similarity used in attention mechanisms with the negative hyperbolic distance, and applies this new attention to the existing Transformer model, graph attention networks (GAT), and Relation Networks (RN). Accordingly, they use Einstein midpoint to compute the aggregation weights of attention in hyperbolic space. The idea of using hyperbolic rather than Euclidean space is based on the assumption that the input embeddings (neural net activations) are on the hyperbolic manifold, which follows power-law distribution and can be seem as a smooth description of tree-like hierarchy of data points. This assumption might hold for small neural networks with relatively low dimensional output. One main reason why this paper adopts the hyperbolic space is that the volume of hyperbolic space grows exponentially with the increase of radius while that of Euclidean space grows only polynomially. Using hyperbolic distance can increase the capacity of networks and handle the complexity of data. Experiments on Transformer and relation network show that Transformer, GAT and RN with the new attention metric produce better performance than Euclidean distance.
Pros:
1. Comparing to the existing methods using representations for shallow models in hyperbolic geometry, this paper extends the idea to deep neural networks. 
2. The proposed attention mechanism can be easily applied to many of existing networks to enhance their capacity.
3. The experiments show several interesting results: 1) hyperbolic recursive transformer (RT) is consistently superior to Euclidean RT across the tasks in this paper; 2) hyperbolic space substantially benefits the low-capacity networks (i.e., low-dimensionality hidden state); 3) Einstein midpoint is better than Euclidean aggregation in hyperbolic space; 4) using sigmoid rather than softmax to compute attention weight may achieve better effectiveness on some tasks for the reason that the attention weights over different entities ay do not compete with each other.
Cons:
1. The novelty of this paper is replacing the Euclidean metric with another existing metric, which has already been used in previous ML models. So the contribution is limited.
2. As explicitly claimed in the paper and also reflected by the experimental results (e.g., Transformer-Big in Table 2). The hyperbolic metric only brings noticeable improvement to small neural nets with limited compacity on relatively small datasets. When applied it to most SOTA models (which are usually large/deep/wide neural networks) on larger datasets, it loses the advantage. This fact might seriously limit the application of the proposed technique.
3. Small models are preferred for inference especially on edge devices. But model compression and knowledge distillation can make small models having similar performance as large models, which might be much better than directly training a small model with the proposed metric.
4. Although hyperbolic metric reflects the power-law distribution, which is a very natural assumption verified on many kinds of real data (social networks and physical statistics), I am not fully convinced that it still holds on an embedding space produced by a neural net (since attention are usually applied to the outputs of a neural net). 
5. In the experiments, does the model with the proposed metric cost similar training/inference time comparing to the baselines? What is the trade-off between improvements and extra time costs? I notice that the results of the proposed attention in Table 2 are ~0.5% higher than the results from the earlier arXiv version of this paper. What is the reason for the improvements? Did you increase the training steps?