Hyperbolic neural networks have shown great potential for modeling complex data. However, existing hyperbolic networks are not completely hyperbolic, as they encode features in the hyperbolic space yet formalize most of their operations in the tangent space (a Euclidean subspace) at the origin of the hyperbolic model. This hybrid method greatly limits the modeling ability of networks. In this paper, we propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model by adapting the Lorentz transformations (including boost and rotation) to formalize essential operations of neural networks. Moreover, we also prove that linear transformation in tangent spaces used by existing hyperbolic networks is a relaxation of the Lorentz rotation and does not include the boost, implicitly limiting the capabilities of existing hyperbolic networks. The experimental results on four NLP tasks show that our method has better performance for building both shallow and deep networks. Our code is released to facilitate follow-up research 1 . Various recent efforts have explored hyperbolic neural networks to learn complex non-Euclidean data properties. Nickel and Kiela (2017); Cvetkovski and Crovella (2016); Verbeek and Suri (2014) learn hierarchical representations in a hyperbolic space and show that hyperbolic geometry  Equal contribution. Corresponding authors. Part of the work was done while Peng Li was working at Tencent. 1 https://github.com/chenweize1998/ fully-hyperbolic-nn can offer more flexibility than Euclidean geometry when modeling complex data structures. After that, Ganea et al. (2018) and Nickel and Kiela (2018) propose hyperbolic frameworks based on the Poincar ball model and the Lorentz model respectively 2 to build hyperbolic networks, including hyperbolic feed-forward, hyperbolic multinomial logistic regression, etc. Encouraged by the successful formalization of essential operations in hyperbolic geometry for neural networks, various Euclidean neural networks are adapted into hyperbolic spaces. These efforts have covered a wide range of scenarios, from shallow neural networks like word embeddings (Tifrea et al., 2018; Zhu et al., 2020), network embeddings (Chami et al., 2019; Liu et al., 2019), knowledge graph embeddings (Balazevic et al., 2019a; Kolyvakis et al., 2019) and attention module (Gul-cehre et al., 2018), to deep neural networks like variational auto-encoders (Mathieu et al., 2019) and flow-based generative models (Bose et al., 2020). Existing hyperbolic neural networks equipped with low-dimensional hyperbolic feature spaces can obtain comparable or even better performance than high-dimensional Euclidean neural networks. Although existing hyperbolic neural networks have achieved promising results, they are not fully hyperbolic. In practical terms, some operations in Euclidean neural networks that we usually use, such as matrix-vector multiplication, are difficult to be defined in hyperbolic spaces. Fortunately for each point in hyperbolic space, the tangent space at this point is a Euclidean subspace, all Euclidean neural operations can be easily adapted into this tangent space. Therefore, existing works (Ganea 2 Both the Poincar ball model and the Lorentz model are typical geometric models in hyperbolic geometry. 5672 et al., 2018; Nickel and Kiela, 2018) formalize most of the operations for hyperbolic neural networks in a hybrid way, by transforming features between hyperbolic spaces and tangent spaces via the logarithmic and exponential maps, and performing neural operations in tangent spaces. However, the logarithmic and exponential maps require a series of hyperbolic and inverse hyperbolic functions. The compositions of these functions are complicated and usually range to infinity, significantly weakening the stability of models. To avoid complicated transformations between hyperbolic spaces and tangent spaces, we propose a fully hyperbolic framework by formalizing operations for neural networks directly in hyperbolic spaces rather than tangent spaces. Inspired by the theory of special relativity, which uses Minkowski space (a Lorentz model) to measure the spacetime and formalizes the linear transformations in the spacetime as the Lorentz transformations, our hyperbolic framework selects the Lorentz model as our feature space. Based on the Lorentz model, we formalize operations via the relaxation of the Lorentz transformations to build hyperbolic neural networks, including linear layer, attention layer, etc. We also prove that performing linear transformation in the tangent space at the origin of hyperbolic spaces (Ganea et al., 2018; Nickel and Kiela, 2018) is equivalent to performing a Lorentz rotation with relaxed restrictions, i.e., existing hyperbolic networks do not include the Lorentz boost, implicitly limiting their modeling capabilities. To verify our framework, we build fully hyperbolic neural networks for several representative scenarios, including knowledge graph embeddings, network embeddings, fine-grained entity typing, machine translation, and dependency tree probing. The experimental results show that our fully hyperbolic networks can outperform Euclidean baselines with fewer parameters. Compared with existing hyperbolic networks that rely on tangent spaces, our fully hyperbolic networks are faster, more stable, and achieve better or comparable results. Hyperbolic geometry is a non-Euclidean geometry with constant negative curvature K . Several hyperbolic geometric models have been applied in previous studies: the Poincar ball (Poincar disk) model (Ganea et al., 2018), the Poincar half-plane model (Tifrea et al., 2018), the Klein model (Gul-cehre et al., 2018) and the Lorentz (Hyperboloid) model (Nickel and Kiela, 2018). All these hyperbolic models are isometrically equivalent, i.e., any point in one of these models can be transformed to a point of others with distance-preserving transformations (Ramsay and Richtmyer, 1995). We select the Lorentz model as the framework cornerstone, considering the numerical stability and calculation simplicity of its exponential/logarithm maps and distance function. Formally, an n -dimensional Lorentz model is the Riemannian manifold L nK = ( L n , g K x ) . K is the constant negative curvature. g K x = diag(  1 , 1 ,    , 1) is the Riemannian metric tensor. Each point in L nK has the form x = [ x t x s ] , x  R n +1 , x t  R , x s  R n . L n is a point set satisfying L n := { x  R n +1 | (cid:104) x , x (cid:105) L = 1 K , x t > 0 } , (cid:104) x , y (cid:105) L :=  x t y t + x (cid:124) s y s = x (cid:124) diag(  1 , 1 ,    , 1) y , where (cid:104) x , y (cid:105) L is the Lorentzian inner product, L n is the upper sheet of hyperboloid (hyper-surface) in an ( n + 1) -dimensional Minkowski space with the origin ( (cid:112)  1 /K, 0 ,    , 0) . For simplicity, we denote a point x in the Lorentz model as x  L nK in the latter sections. The special relativity gives physical interpretation to the Lorentz model by connecting the last n elements x s to space and the 0 -th element x t to time . We follow this setting to denote the 0 -th dimension of the Lorentz model as time axis , and the last n dimensions as spatial axes . Tangent Space Given x  L n K , the orthogonal space of L nK at x with respect to the Lorentzian inner product is the tangent space at x , and is formally written as T x L nK := { y  R n +1 | (cid:104) y , x (cid:105) L = 0 } . Note that T x L nK is a Euclidean subspace of R n +1 . Particularly, we denote the tangent space at the origin as T 0 L nK . Logarithmic and Exponential Maps As shown in Figure 1a, the logarithmic and exponential maps specifies the mapping of points between the hyperbolic space L nK and the Euclidean subspace T x L nK . The exponential map exp K x ( z ) : T x L nK  L nK can map any tangent vector z  T x L nK to L nK by moving along the geodesic  satisfying  (0) = x and  (cid:48) (0) = z . Specifically, the exponential map can be written as exp K x ( z ) = cosh(  ) x + sinh(  ) z ,  =  K (cid:107) z (cid:107) L , (cid:107) z (cid:107) L = (cid:112) (cid:104) z , z (cid:105) L . log K x ( y ) = cosh  1 (  ) (cid:112)  2  1 ( y    = K (cid:104) x , y (cid:105) L . In the special relativity, the Lorentz transformations are a family of linear transformations from a coordinate frame in spacetime to another frame moving at a constant velocity relative to the former. Any Lorentz transformation can be decomposed into a combination of a Lorentz boost and a Lorentz rotation by polar decomposition (Moretti, 2002). Definition 1 (Lorentz Boost) . Lorentz boost describes relative motion with constant velocity and without rotation of the spatial coordinate axes. Given a velocity v  R n (ratio to the speed of light), (cid:107) v (cid:107) < 1 and  = 1  1 (cid:107) v (cid:107) 2 , the Lorentz boost matrices are given by B = (cid:34)    v (cid:124)   v I +  2 1+  vv (cid:124) (cid:35) . Definition 2 (Lorentz Rotation) . Lorentz rotation is the rotation of the spatial coordinates. The Lorentz rotation matrices are given by R = (cid:20) 1 0 (cid:124) 0 R (cid:21) , where R (cid:124) R = I and det( R ) = 1 , i.e., R  SO ( n ) is a special orthogonal matrix. Both the Lorentz boost and the Lorentz rotation are the linear transformations directly defined in the Lorentz model, i.e.,  x  L nK , Bx  L nK and Rx  L nK . Hence, we build fully hyperbolic neural networks on the basis of these two types of transformations in this paper. essential block for neural networks. Although the Lorentz transformations in 2.2 are linear transformations in the Lorentz model, they cannot be directly used for neural networks. On the one hand, the Lorentz transformations transform coordinate frames without changing the number of dimensions. On the other hand, complicated restrictions of the Lorentz transformations (e.g., special orthogonal matrices for the Lorentz rotation) make computation and optimization problematic. Although the restrictions offer nice properties such as spacetime interval invariant to Lorentz transformation, they are unwanted in neural networks. A Lorentz linear layer matrix should minimize the loss while subject to M  R ( m +1)  ( n +1) ,  x  L n , Mx  L m . It is a constrained optimization difficult to solve. We instead re-formalize our lorentz linear layer to learn a matrix M = (cid:2) v (cid:124) W (cid:3) , v  R n +1 , W  R m  ( n +1) satisfying  x  L n , f x ( M ) x  L m , where f x : R ( m +1)  ( n +1)  R ( m +1)  ( n +1) should be a function that maps any matrix to a suitable one for the hyperbolic linear layer. Specifically,  x  L nK , M  R ( m +1)  ( n +1) , f x ( M ) is given as f x ( M ) = f x ( (cid:20) v (cid:124) W (cid:21) ) = (cid:34)  (cid:107) Wx (cid:107) 2  1 /K v (cid:124) x v (cid:124) W (cid:35) , (1) Theorem 1.  x  L nK ,  M  R ( m +1)  ( n +1) , we have f x ( M ) x  L mK . Proof 1. One can easily verify that  x  L nK , we have (cid:104) f x ( M ) x , f x ( M ) x (cid:105) L = 1 /K , thus f x ( M ) x  L mK . Relation to the Lorentz Transformations In this part, we show that the set of matrices { f x ( M ) } defined in Eq.(1) contains all Lorentz rotation and boost matrices. Lemma 1. In the n -dimensional Lorentz model L nK , we denote the set of all Lorentz boost matrices as B , the set of all Lorentz rotation matrices as R . Given x  L nK , we denote the set of f x ( M ) at x without changing the number of space dimension as M x = { f x ( M ) | M  R ( n +1)  ( n +1) } . x  L nK , we have B  M x and R  M x . Considering A = { A  R ( n +1)  ( n +1) |  x  L nK : (cid:104) Ax , Ax (cid:105) L = 1 K , ( Ax ) 0 > 0 } is the set of all valid transformation matrices in the Lorentz model. Then  A = (cid:104) v (cid:124) AWA (cid:105)  A , v A  5674 logmap Tangent Space TO Wx expmap Hence, we can see that A  M x . Since B  A and R  A , therefore B  M x and R  M x . According to Theorem 1 and Lemma 1, both Lorentz boost and rotation can be covered by our linear layer. Relation to the Linear Layer Formalized in the Tangent Space In this part, we show that the conventional hyperbolic linear layer formalized in the tangent space at the origin (Ganea et al., 2018; Nickel and Kiela, 2018) can be considered as a Lorentz transformation with only a special rotation but no boost. Figure 1a visualizes the conventional hyperbolic linear layer. As shown in Figure 1d, we consider a special setting  pseudo-rotation " of our hyperbolic linear layer. Formally, at the point x  L nK , all pseudo-rotation matrices make up the set P x = (cid:8) f x ( (cid:2) w 0 (cid:124) 0 W (cid:3) ) (cid:12)(cid:12) w  R , W  R n  n (cid:9) . As we no longer require the submatrix W to be a special orthogonal matrix, this setting is a relaxation of the Lorentz rotation. Formally, given x  L n K , the conventional hyperbolic linear layer relies on the logarithmic map to map the point into the tangent space at the origin, a matrix to perform linear transformation in the tangent space, and the exponential map to map the final result back to L nK 3 . then we have H x  P x and H x  B = { I } . Proof 3.  x  L nK ,  H  H x , H has the form (cid:2) w 0 (cid:124) 0 W (cid:3) , satisfying (cid:107) Wx s (cid:107) 2  ( wx t ) 2 = 1 K and wx t > 0 . We can verify that f x ( H ) = f x ( (cid:2) w 0 (cid:124) 0 W (cid:3) ) = (cid:20)  (cid:107) Wx s (cid:107) 2  1 /K wxt w 0 (cid:124) 0 W (cid:21) = H . To prove H x  B = I is trivial, we do not elaborate here. Therefore, a conventional hyperbolic linear layer can be considered as a special rotation where the time axis is changed according to the space axes to ensure that the output is still in the Lorentz model. Our linear layer is not only fully 3 Note that Mobius matrix-vector multiplication defined in Ganea et al. (2018) also follows this process 4 The 0 -th dimension of any point in the tangent space at the origin is 0 , therefore the linear matrix has the form diag(  , W ) , where  can be arbitrary number. hyperbolic but also equipped with boost operations to be more expressive. Moreover, without using the complicated logarithmic and exponential maps, our linear layer has better efficiency and stability. A More General Formula Here, we give a more general formula 5 of our hyperbolic linear layer based on f x ( (cid:2) v (cid:124) W (cid:3) ) x , by adding activation, dropout, bias and normalization, y = HL ( x ) = (cid:104)  (cid:107)  ( Wx , v ) (cid:107) 2  1 /K  ( Wx , v ) (cid:105) , (3) where x  L nK , v  R n +1 , W  R m  ( n +1) , and  is an operation function: for the dropout, the function is  ( Wx , v ) = W dropout ( x ) ; for the activation and normalization  ( Wx , v ) =  ( v (cid:124) x + b (cid:48) ) (cid:107) W h ( x )+ b (cid:107) ( W h ( x ) + b ) , where  is the sigmoid function, b and b (cid:48) are bias terms,  > 0 controls the scaling range, h is the activation function. We elaborate  (  ) we use in practice in the appendix. Attention layers are also important for building networks, especially for the networks of Transformer family (Vaswani et al., 2017). We propose an attention module in the Lorentz model. Specifically, we consider the weighted aggregation of a point set P = { x 1 , . . . , x |P| } as calculating the centroid, whose expected (squared) distance to P is minimum, i.e. arg min   L nK (cid:80) |P| i =1  i d 2 L ( x i ,  ) , where  i is the weight of the i -th point. Law et al. (2019) prove that, with squared Lorentzian distance defined as d 2 L ( a , b ) = 2 /K  2 (cid:104) a , b (cid:105) L , the centroid w.r.t. the squared Lorentzian distance is given as  = Centroid (cid:0) {  1 , . . . ,  |P| } , { x 1 , . . . , x |P| } (cid:1) = (cid:80) |P| j =1  j x j  K (cid:12)(cid:12) (cid:107) (cid:80) |P| i =1  i x i (cid:107) L (cid:12)(cid:12) . Given the query set Q = { q 1 , . . . , q |Q| } , key set K = { k 1 , . . . , k |K| } , and value set V = { v 1 , . . . , v |V| } , where |K| = |V| , we exploit the squared Lorentzian distance between points to calculate weights. The attention is defined as 5 Note that this general formula is no longer fully hyperbolic. where n is the dimension of points. Furthermore, multi-headed attention is defined as MHATT ( Q , K , V ) = {  1 , . . . ,  |Q| } , and  i is  i = HL ([  1 i | . . . |  Hi ]) , {  i 1 ,  i 2 , . . . } = ATT i ( HL i Q ( Q ) , HL i K ( K ) , HL i V ( V )) , (6) where H is the head number, [ | . . . | ] is the concatenation of multiple vectors, ATT i (  ,  ,  ) is the i -th head attention, and HL i Q (  ) , HL i K (  ) , HL i V (  ) are the hyperbolic linear layers of the i -th head attention. Other intuitive choices for the aggregation in the Lorentz attention module include Frchet mean (Karcher, 1977) and Einstein midpoint (Un-gar, 2005). The Frchet mean is the classical generalization of Euclidean mean. However, it offers no closed-form solution. Solving the Frchet mean currently requires iterative computation (Lou et al., 2020; Gu et al., 2019), which significantly slows down the training and inference, making it impossible to generalize to deep and large model 6 . On the contrary, Lorentz centroid is fast to compute and can be seen as Frechet mean in pseudo-hyperbolic space (Law et al., 2019). The computation of the Einstein midpoint requires transformation between Lorentz model and Klein model, bringing in numerical instability. The Lorentz centroid we use minimizes the sum of squared distance in the Lorentz model, while the Einstein midpoint does not possess such property in theory. Also, whether the Einstein midpoint in the Klein model has its geometric interpretation in the Lorentz model remains to be investigated, and it is beyond the scope of our paper. Therefore, we adopt the Lorentz centroid in our Lorentz attention. Lorentz Residual The residual layer is crucial for building deep neural networks. Since there is no well-defined vector addition in the Lorentz 6 400 times slower than using Lorentz centroid in our experiment, and no improvement in performance was observed 5676 model, we assume that each residual layer is preceded by a computational block whose last layer is a Lorentz linear layer, and do the residual-like operation within the preceding Lorentz linear layer of the block as a compromise. Given the input x of the computational block and the output o = f ( x ) before the last Lorentz linear layer of the block, we take x as the bias of the Lorentz linear layer. Concretely, the final output of the block is y = (cid:104)  (cid:107)  ( Wo , v , x ) (cid:107) 2  1 /K  ( Wo , v , x ) (cid:105) ,  ( Wo , v , x ) =  ( v (cid:124) o ) (cid:107) W h ( o ) + x s (cid:107) ( W h ( o ) + x s ) , (7) where the symbols have the same meaning as those in Eq.(3). Lorentz Position Encoding Some neural networks require positional encoding for their embedding layers, especially those models for NLP tasks. Previous works generally incorporate positional information by adding position embeddings to word embeddings. Given a word embedding x and its corresponding learnable position embedding p , we add a Lorentz linear layer to transform the word embedding x , by taking the position embedding p as the bias. The overall process is the same as Eq.(7). Note that the transforming matrix in the Lorentz linear layer is shared across positions. This modification gives us one more d  d matrix than the Euclidean Transformer. The increase in the number of parameters is negligible compared to the huge parameters of the whole model. To verify our proposed framework, we conduct experiments on both shallow and deep neural networks. For shallow neural networks, we present results on knowledge graph completion. For deep neural networks, we propose a Lorentz Transformer and present results on machine translation. Dependency tree probing is also done on both Lorentz and Euclidean Transformers to compare their capabilities of representing structured information. Due to space limitations, we report the results of network embedding and fine-grained entity typing experiments in the appendix A. For knowledge graph completion and network embedding, we use our fully hyperbolic linear layer, and for other tasks, we use the general formula given in 3.1, which is a relaxation of our fully hyperbolic linear layer. In the following sections, we denote the models built with our proposed framework as HYBONET . We demonstrate that HyboNet not only outperforms Euclidean and Poincar models on the majority of tasks, but also converges better than its Poincar counterpart. All models in 4.1 are trained with 1 NVIDIA 2080Ti, models in 4.2 are trained with 1 NVIDIA 40GB A100 GPU. We optimize our model with Riemannian Adam (Kochurov et al., 2020). For pre-processing and hyper-parameters of each experiment, please refer to Appendix B. 4.1 Experiments on Shallow Networks In this part, we leverage our Lorentz embedding and linear layers to build shallow neural networks. We show that HyboNet outperforms previous knowledge graph completion models on several popular benchmarks. A knowledge graph contains a collection of factual triplets, each triplet ( h, r, t ) illustrates the existence of a relation r between the head entity h and the tail entity t . Since knowledge graphs are generally incomplete, predicting missing triplets becomes a fundamental research problem. Concretely, the task aims to solve the problem ( h, r, ? ) and ( ? , r, t ). We use two popular knowledge graph completion benchmarks, FB15k-237(Toutanova and Chen, 2015) and WN18RR(Dettmers et al., 2018) in our experiments. We report two evaluation metrics: MRR (Mean reciprocal rank), the average of the inverse of the true entity ranking in the prediction; H@ K , the percentage of the correct entities appearing within the top K positions of the predicted ranking. Setup Similar to Balazevic et al. (2019a), we design a score function for each triplet as s ( h, r, t ) =  d 2 L ( f r ( e h ) , e t ) + b h + b t + , 5677 WN18RR FB15k-237 Model #Dims MRR H@10 H@3 H@1 #Dims MRR H@10 H@3 H@1 TRANSE (Bordes et al., 2013) 180 22.7 50.6 38.6 3.5 200 28.0 48.0 32.1 17.7 DISTMULT (Yang et al., 2015) 270 41.5 48.5 43.0 38.1 200 19.3 35.3 20.8 11.5 COMPLEX (Trouillon et al., 2017) 230 43.2 50.0 45.2 39.6 200 25.7 44.3 29.3 16.5 CONVE (Dettmers et al., 2018) 120 43.5 50.0 44.6 40.1 200 30.4 49.0 33.5 21.3 ROTATE (Sun et al., 2019) 1,000 47.3 55.3 48.8 43.2 1,024 30.1 48.5 33.1 21.0 TUCKER (Balazevic et al., 2019b) 200 46.1 53.5 47.8 42.3 200 34.7 53.3 38.4 25.4 MURP (Balazevic et al., 2019a) 32 46.5 54.4 48.4 42.0 32 32.3 50.1 35.3 23.5 ROTH (Chami et al., 2020a) 32 47.2 55.3 49.0 42.8 32 31.4 49.7 34.6 22.3 ATTH (Chami et al., 2020a) 32 46.6 55.1 48.4 41.9 32 32.4 50.1 35.4 23.6 HYBONET 32 48.9 55.3 50.3 45.5 32 33.4 51.6 36.5 24.4 MURP (Balazevic et al., 2019a)  48.1 56.6 49.5 44.0  33.5 51.8 36.7 24.3 ROTH (Chami et al., 2020a)  49.6 58.6 51.4 44.9  34.4 53.5 38.0 24.6 ATTH (Chami et al., 2020a)  48.6 57.3 49.9 44.3  34.8 54.0 38.4 25.2 HYBONET  51.3 56.9 52.7 48.2  35.2 52.9 38.7 26.3 Table 1: Link prediction results (%) on WN18RR and FB15k-237 in the filtered setting. where e h , e t  L nK are the Lorentz embeddings of the head entity h and the tail entity t , f r (  ) is a Lorentz linear transformation of the relation r and  is a margin hyper-parameter. For each triplet, we randomly corrupt its head or tail entity with k entities and calculate the probabilities for triplets as p =  ( s ( h, r, t )) , where  is the sigmoid function. Finally, we minimize the binary cross entropy loss L =  1 NN (cid:88) i =1   log p ( i ) + k (cid:88) j =1 log(1   p ( i,j ) )   , where p ( i ) and  p ( i,j ) are the probabilities for correct and corrupted triplets respectively, N is the triplet number. We select the model with the best MRR on validation set and report its performance on test set. Results 3.3 shows the results on both datasets. As expected, low-dimensional hyperbolic networks have already achieved comparable or even better results when compared to high-dimensional Euclidean baselines. When the dimensionality is raised to a maximum of 500, HYBONET outperforms all other baselines on MRR, H@3, and H@1 by a significant margin. And as shown in Figures 2a and 2b, HYBONET converges better than other hyperbolic networks on both datasets and has a higher ceiling, demonstrating the superiority of our Lorentz linear layer over conventional linear layer formalized in tangent space. In this part, we build a Transformer (Vaswani et al., 2017) with our Lorentz components introduced in 3. We omit layer normalization for the difficulty of defining hyperbolic mean and variance, but it is still kept in our Euclidean Transformer baseline. In fact,  in Eq.(3) controls the scaling range, which normalize the representations to some extent. We conduct the experiment on two widely-used machine translation benchmarks: IWSLT'14 English-German and WMT'14 English-German. Setup We use OpenNMT (Klein et al., 2017) to build Euclidean Transformer and our Lorentz one. Following previous hyperbolic work (Shimizu et al., 2021), we conduct experiments in low-dimensional settings. To show that our framework can be applied to high-dimensional settings, we additionally train a Lorentz Transformer of the same size as Transformer base, and compare their performance on WMT'14. We select the model with the lowest perplexity on the validation set, and report its BLEU scores on the test set. Results The BLEU scores on the test set of IWSLT'14 and newstest2013 test set of WMT'14 are shown in Table 2. Both Transformer-based hyperbolic models, HYBONET and HATT (Gulcehre et al., 2018), outperform the Euclidean Transformer. However, in HATT , only the calculation of attention weights and the aggregation are performed 5678 IWSLT'14 WMT'14 Model d=64 d=64 d=128 d=256 CONVSEQ 2S EQ 23.6 14.9 20.0 21.8 TRANSFORMER 23.0 17.0 21.7 25.1 HYPER NN++ 22.0 17.0 19.4 21.8 HATT 23.7 18.8 22.5 25.5 HYBONET 25.9 19.7 23.3 26.2 Table 2: The BLEU scores on the test set of IWSLT'14 and WMT'14 under the low-dimensional setting. in hyperbolic space, leaving the remaining computational blocks in the Euclidean space. That is, HATT is a partially hyperbolic Transformer. As a result, the merits of hyperbolic space are not fully exploited. On the contrary, HYBONET performs all its operations in the hyperbolic space, thus better utilizes the hyperbolic space, and achieve significant improvement over both Euclidean and partially hyperbolic Transformer. Apart from the low-dimensional setting that is common in hyperbolic literature, we scale up the model to be the same size as Transformer base (512-dimensional input) (Vaswani et al., 2017). We report the results in Table 3. HYBONET outperforms TRANSFORMER and HATT with the same model size, and is very close to the much bigger TRANSFORMER big . In this part, we verify the superiority of HYBONET in capturing latent structured information in unstructured sentences through dependency tree probing. It has been shown that neural networks implicitly embed syntax trees in their intermediate context representations (Hewitt and Manning, 2019; Raganato et al., 2018). One reason we think HYBONET performs better in machine translation is that it better captures structured information in the sentences. To validate this, we perform a probing on TRANSFORMER , HATT and HYBONET obtained in 4.2.1. We use dependency parsing result of stanza (Qi et al., 2020) on IWSLT'14 English Distance Depth Model UUAS Dspr. corpus as our dataset. The data partition is kept. Setup For a fair comparison, we probe all the models in hyperbolic space following Chen et al. (2021). Four metrics are reported: UUAS (undi-rected attachment score), the percentage of undirected edges placed correctly against the gold tree; Root% , the precision of the model predicting the root of the syntactic tree; Dspr. and Nspr. , the Spearman correlations between true and predicted distances for each word in each sentence, true depth ordering and the predicted ordering, respectively. Please refer to the appendix for details. Results The probing results are shown in Table 2. HYBONET outperforms other baselines by a large margin. Obviously, syntax trees can be better reconstructed from the intermediate representation of HYBONET 's encoder, which shows that HYBONET better captures syntax structure. The result of HATT is also worth noting. Because HATT is a partially hyperbolic Transformer, intuitively, its ability to capture the structured information should be better than Euclidean Transformer, but worse than HYBONET . Our result confirms this suspicion indeed. The probing on HATT indicates that as the model becomes more hyperbolic, the ability to learn structured information becomes stronger. Hyperbolic geometry has been widely investigated in representation learning in recent years, due to its great expression capacity in modeling complex data with non-Euclidean properties. Previous works have shown that when handling data with hierarchy, hyperbolic embedding has better representation capacity and generalization ability (Cvetkovski and Crovella, 2016; Verbeek and Suri, 2014; Walter, 2004; Kleinberg, 2007; Krioukov et al., 2009; Cvetkovski and Crovella, 2009; Shavitt and Tankel, 2008; Sarkar, 2011). Moreover, Ganea et al. (2018) and Nickel and Kiela (2018) introduce the basic operations of neural networks in the Poincar ball 5679 and the Lorentz model respectively. After that, researchers further introduce various types of neural models in hyperbolic space including hyperbolic attention networks (Gulcehre et al., 2018), hyperbolic graph neural networks (Liu et al., 2019; Chami et al., 2019), hyperbolic prototypical networks (Mettes et al., 2019) and hyperbolic capsule networks (Chen et al., 2020). Recently, with the rapid development of hyperbolic neural networks, people attempt to utilize them in various downstream tasks such as word embeddings (Tifrea et al., 2018), knowledge graph embeddings (Chami et al., 2020b), entity typing (Lpez et al., 2019), text classification (Zhu et al., 2020), question answering (Tay et al., 2018) and machine translation (Gulcehre et al., 2018; Shimizu et al., 2021), to handle their non-Euclidean properties, and have achieved significant and consistent improvement. Our work not only focus on the improvement in the downstream tasks that hyperbolic space offers, but also show that hyperbolic linear transformation used in previous work is just a relaxation of Lorentz rotation, giving a different theoretical interpretation for the hyperbolic linear transformation. In this work, we propose a novel fully hyperbolic framework based on the Lorentz transformations to overcome the problem that hybrid architectures of existing hyperbolic neural networks relied on the tangent space limit network capabilities. The experimental results on several representative NLP tasks show that compared with other hyperbolic networks, HYBONET has faster speed, better convergence, and higher performance. In addition, we also observe that some challenging problems require further efforts: (1) Though we have verified the effectiveness of fully hyperbolic models in NLP, exploring its applications in computer vision is still a valuable direction. (2) Though HYBONET has better performance on many tasks, it is slower than Euclidean networks. Also, because of the floating-point error, HYBONET cannot be sped up with half precision training. We hope more efforts can be devoted into this promising field. This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute for Guo Qiang at Tsinghua University, Beijing Academy of Artificial Intelligence (BAAI), and
abstain abstain abstain objective method result method abstain abstain abstain abstain other abstain abstain other abstain abstain method abstain abstain abstain abstain objective objective objective method result result result abstain abstain abstain method abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain method other abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other other other other other abstain objective abstain objective abstain abstain abstain other