Although neural tensor networks (NTNs) have been successful in many natural language processing tasks, they require a large number of parameters to be estimated, which often results in overfitting and long training times. We address these issues by applying eigendecompo-sition to each slice matrix of a tensor to reduce the number of parameters. We evaluate our proposed NTN models in two tasks. First, the proposed models are evaluated in a knowledge graph completion task. Second, a recursive NTN (RNTN) extension of the proposed models is evaluated on a logical reasoning task. The experimental results show that our proposed models learn better and faster than the original (R)NTNs. Alongside the nonlinear activation functions, linear mapping by matrix multiplication is an essential component of neural network (NN) models, as it determines the feature interaction and thus the expressiveness of models. In addition to the matrix-based mapping, neural tensor networks (NTNs) (Socher et al., 2013a) employ a 3-dimensional tensor to capture direct interactions among input features. Due to the large expressive capacity of 3D tensors, NTNs have been successful in an array of natural language processing (NLP) and machine learning tasks, including knowledge graph completion (KGC) (Socher et al., 2013a), sentiment analysis (Socher et al., 2013b), and reasoning with logical semantics (Bowman et al., 2015). However, since a 3D tensor has a large number of parameters, NTNs need longer time to train than other NN models. Moreover, the millions of parameters often make the model suffer from overfitting (Yang et al., 2015). techniques drastically decrease the number of parameters in an NTN without diminishing its expressiveness. We use the matrix decomposition techniques that are utilized for KGC in Yang et al. (2015) and Trouillon et al. (2016). Yang et al. (2015) imposed a constraint that a matrix in the bilinear term in their model had to be diagonal. As mentioned in a subsequent section, this is essentially equal to assuming that the matrix be symmetric and performing eigendecomposition. Trouillon et al. (2016) also applied eigendecom-position to a matrix by regarding it as the real part of a normal matrix. Following these studies, we perform simultaneous diagonalization on all slice matrices of a NTN tensor. As a result, mapping by a 3D ( n  n  k ) tensor is replaced with an array of k triple inner products of two input vectors and a weight vector. Thus, we obtain two new NTN models where the number of parameters is reduced from O ( n 2 k ) to O ( nk ) . On a KGC task, these parameter-reduced NTNs (NTN-Diag and NTN-Comp) alleviate overfitting and outperform the original NTN. Moreover, our proposed NTNs can learn faster than the original NTN. We also show that our proposed models perform better and learn faster in a recursive setting by examining a logical reasoning task. We consider mapping in a neural network (NN) layer that takes two vectors as input, such as recursive neural networks. Recurrent neural networks also has this structure, with one input vector being the hidden state from the previous time step. As a mapping before activation in the NN layer, linear mapping (matrix multiplication) is commonly used: W 1 x 1 + W 2 x 2 = [ W 1 , W 2 ] [ x 1 x 2 ] = W x . Here, since x 1 , x 2  R n , W 1 , W 2  R k  n , this linear mapping is a transformation from R 2 n to R k . Linear mapping, which is a standard component of NNs, has been applied successfully in many tasks. However, it cannot consider the interaction between different components of two input vectors, which renders it not ideal for modeling complex compositional structures such as trees and graphs. To alleviate this problem, some models such as NTNs (Socher et al., 2013a) have explored 3D tensors to yield more expressive mapping: x T1 W [1: k ] x 2 =   x T1 W [1] x 2 x T1 W [2] x 2 ... x T1 W [ k ] x 2   =   sum ( W [1]  ( x 1  x 2 ) ) sum ( W [2]  ( x 1  x 2 ) ) . The output of this mapping is an array of k bilinear products in the form of x T1 W [ i ] x 2 . Thus, this is also a transformation from R 2 n to R k . Each element of the output of this mapping equals the sum of W [ i ]  ( x 1  x 2 ) , where  and  represent, respectively, the Hadamard and the outer products. Hence this mapping captures the direct interaction between different components (or features) in two input vectors. Thanks to this expressiveness, NTNs are effective in tasks such as knowledge graph completion (Socher et al., 2013a), sentiment analysis (Socher et al., 2013b), and logical reasoning (Bowman et al., 2015). Although mapping by a 3D tensor provides expressiveness, it has a large number ( O ( n 2 k ) ) of parameters. Due to this, NTNs often suffer from overfitting and long training times. To reduce the number of parameters of a slice matrix W [ i ]  R n  n in a tensor, simple matrix decomposition (SMD) is commonly used (Bai et al., 2009). SMD factorizes W [ i ] into a product of two low rank matrices S [ i ]  R n  m and T [ i ]  R m  n ( m  n ): W [ i ]  S [ i ] T [ i ] . By plugging (1) into bilinear term x T1 W [ i ] x 2 , we obtain the approximation x T1 S [ i ] T [ i ] x 2 . SMD reduces the number of parameters of W [ i ] from n 2 to 2 nm . However, the dimension m for S and T is a hyperparameter and must be determined prior to training. This section introduces two techniques that can simultaneously diagonalize all slice matrices W [1] , . . . , W [ i ] , . . . , W [ k ]  R n  n . As described in (Liu et al., 2017), we make use of the fact that if matrices V [1: k ] form a commuting family: i.e., V [ i ] V [ j ] = V [ j ] V [ i ] ,  i, j  { 1 , 2 , . . . , k } , they can be diagonalized by a shared orthogonal or unitary matrix. Both of the two techniques reduce the number of parameters of W [ i ] to O ( n ) from O ( n 2 ) . Many NLP datasets contain symmetric patterns. For example, if binary relation (Bob, is relative of, Alice) holds in a knowledge graph, then (Alice, is relative of, Bob) should also hold in it. English phrases dog and cat and cat and dog have identical meaning. For symmetric structures, we can reasonably suppose that each slice matrix W [ i ] of a 3D tensor is symmetric because x T1 W [ i ] x 2 must equal x T2 W [ i ] x 1 . When W [ i ]  R n  n is symmetric, it can be diagonalized as: W [ i ] = O [ i ] W [ i ]  O [ i ]T where O [ i ]  R n  n is an orthogonal matrix and W [ i ]   R n  n is a diagonal matrix. Note that an orthogonal matrix O [ i ] may not be equal to O j if i  = j . However, if all of the slice matrices W [1] , . . . , W [ i ] , . . . , W [ k ]  R n  n are commuting, we can diagonalize every slice matrix with the same orthogonal matrix O . By substituting W [ i ] with OW [ i ]  OT into bilinear term x T1 W [ i ] x 2 , we can rewrite it as follows: x T1 W [ i ] x 2 = x T1 OW [ i ]  OT x 2 = y T1 W [ i ]  y 2 =  y 1 , w [ i ] , y 2  (2) where y 1 = OT x 1 , y 2 = OT x 2 , w [ i ] = diag ( W [ i ]  )  R n and  a , b , c  denotes a triple inner product defined by  a , b , c  =  nl =1 a l b l c l . This reduces the number of parameters in a single slice matrix from n 2 to n . Since most of the structures in the NLP data are not symmetric, the symmetric matrix assumption is usually violated. To obtain more expressive diagonal matrix, we regard each slice matrix W [ i ] as the real part of a complex matrix and consider its eigendecomposition. For any real matrix W [ i ] , there exists a complex normal matrix Z [ i ] whose real part is equal to it: W [ i ] =  ( Z [ i ] ) . (  ) represents an operation that takes the real part of a complex number, vector or matrix. Further, any complex normal matrix can be diagonalized by a unitary matrix. With these two properties, any real matrix W [ i ] can be diagonalized as follows (Trouillon et al., 2016): W [ i ] =  ( Z [ i ] ) =  ( U [ i ] Z [ i ]  U [ i ]  ) . Here, U [ i ]  C n  n is a unitary matrix, Z [ i ]   C n  n is a diagonal matrix, and U [ i ]  is the conjugate transpose of U [ i ] . To guarantee that every slice matrix can be diagonalized with the same unitary matrix U instead of U [ i ] , we assume all of the normal matrices Z [1] , . . . , Z [ i ] , . . . , Z [ k ]  C n  n are commuting as in Section 3.2.1. Substituting  ( UZ [ i ]  U  ) whose U is the same unitary matrix in all slice matrices, we can rewrite every bilinear term x T1 W [ i ] x 2 as follows: x T1 W [ i ] x 2 =  (  y 1 , w [ i ] , y 2  ) =  ( y 1 ) ,  ( w [ i ] ) ,  ( y 2 )  +  ( y 1 ) ,  ( w [ i ] ) ,  ( y 2 )  +  ( y 1 ) ,  ( w [ i ] ) ,  ( y 2 )    ( y 1 ) ,  ( w [ i ] ) ,  ( y 2 )  , (3) where y 1 = UT x 1 , y 2 = U  x 2 , w [ i ] = diag ( Z [ i ]  )  C n , and  y 1 , w [ i ] , y 2  is the triple Hermitian inner product of y 1 , w [ i ] and y 2 defined by  a , b , c  =  nl =1 a l b l c l . This technique reduces the number of parameters of the matrices from n 2 to 2 n . As shown in the right-hand side of Eq. (3),  (  y 1 , w [ i ] , y 2  ) can be replaced with three additions and a subtraction of the triple inner product of real vectors. This section introduces the baseline and our proposed models. After describing them, we explain how to extend them for handling compositional structures like binary trees. First, we describe a standard single layer neural network (NN) model for two vectors x 1 , x 2  R n . The model uses linear mapping V  R k  2 n to combine two input vectors: f ( V [ x 1 x 2 ] + b ) where b  R k is a bias term and f is a non-linear activation function. The NN model has only (2 n + 1) k parameters, and does not consider the direct interactions between x 1 and x 2 . Socher et al. (2013a) proposed a neural tensor network (NTN) model that uses a 3D tensor W [1: k R n  n  k to combine two input vectors: Unlike the standard NN model, NTN can directly relate two input vectors using a tensor. However, it has too many parameters; ( n 2 + 2 n + 1) k . Although the NTN model has tremendous expressive power, it is extremely time-consuming to compute, since a naive 3D tensor product incur O ( n 2 k ) computation time. To overcome this weakness, Zhao et al. (2015) and Liu et al. (2015) independently introduced simple matrix decomposition (SMD) to the NTN model by replacing each slice matrix W [ i ] with its factorized approximation given by Eq. (1): f ( x T1 S [1: k ] T [1: k ] x 2 + V [ x 1 x 2 ] + b ) where S [1: k ]  R n  m  k , T [1: k ]  R m  n  k . When m  n , the NTN-SMD model drastically reduces the number of parameters compared to the original NTN model; i.e., from ( n 2 + 2 n + 1) k to (2 mn + 2 n + 1) k . In this paper, we introduce two new NTN models: NTN-Diag and NTN-Comp, both of which reduce the number of parameters in a 3D tensor more than NTN-SMD with little loss in the model's generalization performance. Table 1 summarizes the number of parameters in each model. We replace all slice matrices W [ i ] of W [1: k ] with the triple inner product formulation of Eq. (2) by assuming that they are symmetric and commuting. As a result, we derive the following new NTN formulation: f (    x 1 , w [1] , x 2  ...  x 1 , w [ k ] , x 2    + V [ x 1 x 2 ] + b ) where w [ i ]  R n ,  i  { 1 , 2 , . . . , k } . Thus, under the symmetric and commuting matrix constraints, we regard mapping by a 3D tensor as an array of k triple inner products. The total number of parameters is just (3 n + 1) k . By assuming that W [1] , . . . , W [ i ] , . . . , W [ k ] are real parts of normal matrices forming a commuting family, we can replace each slice matrix of a tensor term in NTN with the triple Hermitian inner product shown in Eq. (3): f (    (  x 1 , w [1] , x 2  ) ...  (  x 1 , w [ k ] , x 2  )   +  ( V [ x 1 x 2 ]) + b ) where x 1 , x 2  C n , V  C n  n and w [ i ]  C n ,  i  { 1 , 2 , . . . , k } . Similar to NTN-Diag, we regard mapping by a 3D tensor as an array of k triple Hermitian inner products. The total number of parameters is just (6 n + 1) k . As is clear of its form, NTN-Diag is a special case of NTN-Comp whose vectors x 1 , x 2 and w [ i ] are constrained to be real. We extend the above NTN models to handle compositional structures. As a representative of compositional structures, we consider a binary tree where each NTN layer computes a vector representation for a node by combining two vectors from its child nodes in the lower layer. Except for NTN-Comp, the models implement mappings R n  R k so that each of their layers can receive its lower layer's output directly, if k equals to n . Thus, the models do not have to be modified for them. However, NTN-Comp cannot receive its lower layer's output as it is because NTN-Comp is a mapping from C n to R k . To solve this problem, we set k to 2 n and treat the output y   R 2 n as the concatenation of vectors representing the real and imaginary parts of y  C n :  ( y ) = ( y  1 ,    , y  n ) ,  ( y ) = ( y  n +1 ,    , y  2 n ) . Note that this approach is valid since Eq. (3) can actually be defined in real vector space by transforming the complex vectors in C n into real vectors in R 2 n . In KGC, researchers usually design scoring function  for the given triplet ( s, r, o ) to judge whether it is a fact or not. Here ( s, r, o ) denotes that entity s is linked to entity o by relation r . RESCAL (Nickel et al., 2011) uses e T s W r e o as  , where e s , e o are entity embedding vectors and W r is an embedding matrix of relation r . This bilinear operation is effective for the task, but its computational cost is high and it suffers from overfitting. To overcome these problems, DistMult (Yang et al., 2015) adopts the triple inner product  e s , w r , e o  as  , where w r is an embedding vector of relation r . This solves those problems, but it degrades the model's ability to capture directionality of relations, because the scoring function of DistMult is symmetric with respect to s and o ; i.e.,  e s , w r , e o  =  e o , w r , e s  . To reconcile the complexity and expressiveness of a model, ComplEx (Trouillon et al., 2016) uses complex vectors for entity and relation embeddings. As scoring function  , they adopted the triple Hermitian inner product  (  e s , w r , e o  ) , where e o denotes the complex conjugate of e o . Since  (  e s , w r , e o  )  =  (  e o , w r , e s  ) , ComplEx solves the expressiveness problem of DistMult without full matrices as relation embeddings. We can regard DistMult as a special case of RESCAL with a symmetric matrix constraint on W r . ComplEx is also a RESCAL variant with W r as the real part of a normal matrix. Our research is based on these works, but to the best of our knowledge, no previous work applied this ap-509 proach to reduce the number of parameters in a tensor. To give additional expressiveness power to standard (R)NNs, many architectures have been proposed, such as LSTM (Hochreiter and Schmid-huber, 1997), GRU (Cho et al., 2014), and CNN (LeCun et al., 1998). NTN (Socher et al., 2013a) and RNTN (Socher et al., 2013b) are other such architectures. However, (R)NTNs differ in that they only add 3D tensor mapping to standard neural networks. Thus, they can also be regarded as a powerful basic component of NNs because 3D tensor mapping can be applied to more complicated architectures such as those examples. Several researchers reduced the number of parameters of NNs by using specific parameter sharing mechanisms. Cheng et al. (2015) used circulant matrix mapping instead of conventional linear mapping and improved the time complexity of the matrix-vector product by using Fast Fourier Transformation (FFT). Circulant matrix C ( w ) =   w 1 w n . . . w 3 w 2 w 2 w 1 . . . w 4 w 3 ... ... ... ... ... w n  1 w n  2 . . . w 1 w n w n w n  1 . . . w 2 w 1   for w T = ( w 1 , . . . , w n ) can be factorized into F  1 diag ( F w ) F with the Fourier matrix F . By assuming each slice matrix W [ i ] of W [1: k ] is circulant, we get the same scoring function as that in Eq. (3); x T1 W [ i ] x 2 = x T1 F  1 diag ( F w [ i ] ) F x 2 =  (  x  1 , w [ i ]  , x 2   ) where x  1 = F x 1 , x  2 = F x 2 , and w [ i ]  = 1 n diag ( F w [ i ] ) are complex vectors in C n . In this sense, NTN-Comp is equivalent to NTN where slice matrices of the 3D tensor are restricted to be circulant. Hayashi and Shimbo (2017) established a more detailed proof of the equivalence. Lu et al. (2016) employed a Toeplitz-like structured matrix, reducing parameters of LSTM. Chen et al. (2015) used a feature hashing technique to reduce parameters in RNN. Although these techniques can also be extended to reduce the number of tensor-related parameters in NTN, the former needs FFT operations; i.e., O ( n log n ) computation time, and the latter's contribution is only a reduction in memory consumption. To evaluate their performance for link prediction on knowledge graphs, we compared our proposed methods (NTN-Diag and NTN-Comp) to baseline methods (NTN (Socher et al., 2013a) and NTN-SMD). Let E and R denote entities and relations, respectively. A relational triplet , or simply a triplet , ( s, r, o ) is a triple with s, o  E and r  R . It represents a proposition that relation r holds between subject entity s and object entity o . A triplet is called a fact if the proposition it denote is true. A knowledge graph is a collection of knowledge triplets, with the understanding that all its mem-ber triplets are facts. It is called a graph because each triplet can be regarded as an edge in a directed graph; the vertices in this graph represent entities in E , and each edge is labeled by a relation in R . Let G be a knowledge graph, viewed as a collection of facts. Knowledge graph completion (KGC) is the task of predicting whether unknown triplet ( s  , r  , o  )  G such that s  , o   E , r   R is a fact or not. The standard approach to KGC is to design a score function  : E  R  E  R that assigns a large value when a triplet seems to be a fact. Socher et al. (2013a) defined it as follows. Here, e s , e o  R n are entity embeddings and W r , V r , b r , u r are parameters for each relation r . u r is a k -dimensional vector to map f 's output R k to R which indicates a score. f is the hyperbolic tangent. To compare the performances of the baselines and proposed models, we change the mapping before an activation. For NTN-SMD, we change term e T s W [1: k ] r e o to e T s S [1: k ] r T [1: k ] r e o . To apply NTN-Diag and NTN-Comp in this model, 510 WN18 FB15K MRR Hits@ MRR Hits@ model Filter Raw 1 3 10 Filter Raw 1 3 10 NN 0.111 0.106 7.0 11.7 18.3 0.259 0.165 17.9 28.1 41.7 NTN ( k = 1 ) 0.740 0.512 67.6 78.4 85.2 0.347 0.188 24.1 39.3 55.2 NTN ( k = 4 ) 0.754 0.530 69.3 79.5 86.3 0.380 0.198 27.1 43.0 59.2 NTN-SMD ( m = 1 ) 0.243 0.216 15.9 26.1 40.9 0.278 0.172 19.3 30.1 44.7 NTN-SMD ( m = 2 ) 0.224 0.199 15.1 23.8 37.2 0.298 0.177 20.7 32.7 47.8 NTN-SMD ( m = 3 ) 0.299 0.255 20.4 32.4 49.2 0.312 0.183 21.7 34.5 49.9 NTN-SMD ( m = 10 ) 0.533 0.413 42.2 59.4 74.5 0.333 0.188 22.8 37.5 53.8 NTN-SMD ( m = 25 ) 0.618 0.463 52.1 67.8 80.0 0.341 0.187 23.2 38.6 55.5 NTN-Diag 0.824 0.590 74.8 89.6 92.7 0.443 0.238 31.5 51.2 68.5 NTN-Comp 0.857 0.610 80.1 90.9 93.1 0.490 0.246 36.3 56.7 71.9 DistMult  0.822 0.532 72.8 91.4 93.6 0.654 0.242 54.6 73.3 82.4 ComplEx  0.941 0.587 93.6 94.5 94.7 0.692 0.242 59.9 75.9 84.0 Table 3: Mean Reciprocal Rank (MRR) and Hits@n for the models tested on WN18 and FB15k. The loss function used to train the models is shown below: N  i =1 C  c =1 max ( 0 , 1   ( T ( i ) ) +  ( T ( i ) c )) +     22 , where     22 is an L2 regularization term, T ( i ) denotes the i -th example of training data of size N , and T ( i ) c is one of C randomly sampled negative examples for the i -th training example. We generated negative samples of a triplet ( s, r, o ) by corrupting its subject or object entity. We used the Wordnet (WN18) and Freebase (FB15k) datasets to verify the benefits of our proposed methods. The dataset statistics are given in Table 2. We selected hyper-parameters based on Socher et al. (2013a) and Yang et al. (2015): For all of the models, the size of mini-batches was set to 1000 , the dimensionality of the entity vector to d = 100 , and the regularization parameter to 0 . 0001 ; the tensor slice size was set to k = 4 for all models, except NTN for which we also tested with k = 1 to see the influence of the slice size on the performance. We performed 300 epochs of training for Wordnet and 100 on Freebase using Adagrad (Duchi et al., 2011) with the initial learning rate set to 0 . 1 . For evaluation, we removed the subject or object entity of each test example and then replaced it with all the entities in E . We computed the scores of these corrupted triplets and ranked them in descending order of scores. We here report the results collected in filtered and raw settings. In the filtered setting, given test example ( s, r, o ) , we remove from the ranking all the other positive triplets that appear in either training, validation, or test dataset, whereas the raw metrics do not remove these triplets. Experimental results are shown in Table 3. We observe the following:  The performance of NN and NTNs differs considerably; Apparently, NN is inadequate for this task. By comparing the results of NTNs with different slice sizes, we see that k = 4 performs better than k = 1 . NTN-SMDs perform better than NN, but are all inferior to NTNs, although their results improved as m (the rank of decomposed matrices) is increased. NTN-Diag achieved better results than NTN, although it has far fewer parameters than NTN and the datasets contain many unsymmetrical triplets. This demonstrates that NTN-Diag solves the overfitting problem of NTN without sacrificing the expressiveness power. NTN-Diag also has fewer parameters than the smallest ( m = 1 ) NTN-SMD. Thus, 511 Conjunctive normal form m  i =1 n i  j =1 A ij Disjunctive normal form m  i =1 n i  j =1 A ij Table 4: Conjunctive and disjunctive normal forms in propositional logic. we conclude that NTN-Diag is a better alternative of NTN than NTN-SMD is, in terms of both accuracy and computational cost. NTN-Comp outperformed NTN-Diag, showing that its flexible constraint on matrices yielded additional expressiveness. However, NTN-Diag and NTN-Comp do not exceed DistMult and ComplEx, respectively, in almost all measures. To validate the performance of our proposed models in a recursive neural network setting, we experimentally tested them by having them solve a semantic compositionality problem in logic. This task definition basically follows Bowman et al. (2015): Given a pair of artificially generated propositional logic formulas, classify the relation between the formulas into one of the seven basic semantic relations of natural logic (MacCartney and Manning, 2009). Table 5 shows these seven relation types. The formulas consist of propositional variables, negation, and conjunction and disjunction connectives. Although Bowman et al. (2015) generated formulas with no constraint on its form, we restricted them to disjunctive normal not p 3  p 3 p 3  ( p 3 or p 2 ) ( p 1 or ( p 2 or p 4 )))  ( p 2 and not p 4 ) Table 6: Short examples of type of formulas and their relations in datasets. form (DNF) or conjunctive normal form (CNF) (Table 4). Recall that any propositional formula can be transformed into these forms. Following Bowman et al. (2015), we constructed a model that infers the relations between formula pairs, as described in Table 6. The model consists of two layers: composition and comparison layers (Figure 1). The composition layer outputs the embeddings of both left and right formulas by recursive neural networks. Subsequently, the comparison layer compares the two embeddings using a single layer neural network, and then a softmax classifier receives its output. In the composition layer, we set different parameters for and and or operations. As a loss function, we used cross entropy with L2 regularization and apply the NTNs in Section 4 to the comparison layer and uses RNTNs for as the composition layer. In this experiment, an example is a pair of propositional formulas, and its class label is the seven relation types between the pair. We generated examples following the protocol described in Bowman et al. (2015), with the exception that the formulas are restricted to CNF or DNF, as mentioned above. We obtained 62,589 training examples, 13,413 validation examples, and 55,150 test examples. Each formula in the training and validation examples contains up to four logical operators, whereas those in the test examples have 512 Model 1 2 3 4 5 6 7 8 9 10 11 12 Avg. up to 12 logical operators. Every formula consists of up to four variables taken from six propositional variables that are shared among all the examples. Hyperparameters and optimization are based on Bowman et al. (2015): Embedding size d = 25 (for RNN, d = 45 ) and the output size of comparison layer is k = 75 , and we used AdaDelta (Zeiler, 2012) for an optimizer. We searched for the best coefficient  of L2 regularization in   { 0 . 0001 , 0 . 0003 , 0 . 0005 , 0 . 0007 , 0 . 0009 , 0 . 001 } , whereas Bowman et al. (2015) set  to 0 . 001 for RNN and 0 . 0003 for RNTN. The results are shown in Table 7. From the table, we observe the following:  As with KGC, the large difference in performance between RNN and RNTN suggests that this logical reasoning task requires feature interactions to be captured 1 . 1 Bowman (2016) also evaluated TreeLSTM, but its advantage over RNN was unclear in their experiment. For that  RNTN-Diag achieved the best accuracy except for Tests 2 and 12 and outperformed RNTN except for Test 2. This is not surprising because both and and or are symmetric: p 1 and p 2 equals p 2 and p 1 . This matches the tensor term in RNTN-Diag which is symmetric with respect to x 1 and x 2 . RNTN-Comp was the second best except for Tests 13 and 1012. For all tests, its accuracy was comparable with or superior to that of RNTN. RNTN-SMD ( m = 1 ) was inferior to RNTN for most test sets, although some good results were observed with m = 1 , 2 , 3 on Tests 11 and 12. Indeed, except for Tests 9 12, RNTN-SMD ( m = 1 ) was inferior even to RNN despite the larger number of parameters in RNTN-SMD. RNTN-SMD ( m = 2 ) obtained better results than m = 1 , but it is still worse than RNTN except for Tests 10-12. Further increase in m ( m = 4 , 8 , 16 ) worsened the accuracy despite an increase of the number of parameters. We also evaluated the stability of the model over different trials and hyperparameters. Table 8 shows the best average accuracy for each compared model (among all the tested  ) on the validation set. The parenthesized figures (on the rightmost column) show the standard deviation over five independent trials used for computing the average, i.e., all five trials used the same  value that achieved the best average accuracy. We see that RNTN-SMDs have larger standard deviations than reason, we did not test TreeLSTM in this paper. RNTN, RNTN-Diag and RNTN-Comp. This indicates that RNTN-SMD is a less reliable model. RNTN-SMDs are also unstable, not only within the same  , but also between different  s. Figure 2 describes how accuracies are impacted by  s. The top graph shows validation accuracies between different  values. RNTN, RNTN-Diag and RNTN-Comp are stable, whereas RNN and RNTN-SMDs have steep drops. The bottom one describes the accuracies for Test 12. This also shows that RNTN-SMDs are unstable and that RNTN-Diag achieves distinctive performances. Finally, Figure 3 shows that training times increase quadratically with dimension for RNTN that has O ( n 2 k ) parameters, but not for our methods, which have only O ( nk ) parameters. We proposed two new parameter reduction methods for tensors in NTNs. The first method constrains the slice matrices to be symmetric, and the second assumes them to be normal matrices. In both methods, the number of a 3D tensor param-Figure 3: Training times of the models. eters is reduced from O ( n 2 k ) to O ( nk ) after the constrained matrices are eigendecomposed. By removing the tensor's surplus parameters, our methods learn better and faster as was shown in experiments. 2 Future work will test the versatility of our proposals, RNTN-Diag and RNTN-Comp, in other tasks that deal with data sets exhibiting carious structures.
abstain method objective abstain abstain objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain objective abstain objective objective method other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other method method abstain method method method abstain method method method method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain method method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result objective abstain abstain abstain result abstain