The explicit use of syntactic information has been proved useful for neural machine translation (NMT). However, previous methods resort to either tree-structured neural networks or long linearized sequences, both of which are inefficient. Neural syntactic distance (NSD) enables us to represent a constituent tree using a sequence whose length is identical to the number of words in the sentence. NSD has been used for constituent parsing, but not in machine translation. We propose five strategies to improve NMT with NSD. Experiments show that it is not trivial to improve NMT with NSD; however, the proposed strategies are shown to improve translation performance of the baseline model (+2.1 (EnJa), +1.3 (Ja En), +1.2 (EnCh), and +1.0 (ChEn) BLEU). In recent years, neural machine translation (NMT) has been developing rapidly and has become the de facto approach for machine translation. To improve the performance of the conventional NMT models (Sutskever et al., 2014; Bahdanau et al., 2014), one effective approach is to incorporate syntactic information into the encoder and/or decoder of the baseline model. Based on how the syntactic information is represented, there are two categories of syntactic NMT methods: (1) those that use tree-structured neural networks (NNs) to represent syntax structures (Eriguchi et al., 2016; Hashimoto and Tsuruoka, 2017), and (2) those that use linear-structured NNs to represent linearized syntax structures (Li et al., 2017; Ma et al., 2017, 2018). For the first category, there is a direct corresponding relationship between the syntactic structure and the NN structure, but the complexity of NN structures usually makes training inCorresponding author efficient. In contrast, for the second category, syntactic structures are linearized and represented using linear-structured recurrent neural networks (RNNs), but the linearized sequence can generally be quite long and therefore training efficiency is still a problem. Although using a shorter sequence may improve the efficiency, some syntactic information is lost. We propose a method of using syntactic information in NMT that overcomes the disadvantages of both methods. The basis of our method is the neural syntactic distance (NSD), a recently proposed concept used for constituent parsing (Shen et al., 2018; Gomez-Rodrguez and Vilares, 2018). NSD makes it possible to represent a constituent tree as a sequence whose length is identical to the number of words in the sentence (almost) without losing syntactic information. However, there are no previous studies that use NSD in NMT. Moreover, as demonstrated by our experiments, using NSD in NMT is far from straightforward, so we propose five strategies and verify the effects empirically. The strategies are summarized below. Extend NSD to dependency trees, which is inspired by the dependency language model (Shen et al., 2010). Use NSDs as input sequences 1 , where an NSD is regarded as a linguistic input feature (Sennrich and Haddow, 2016). Use NSDs as output sequences, where the NMT and prediction of the NSD are simultaneously trained through multi-task learning (Firat et al., 2016). Use NSD as positional encoding (PE), which is a syntactic extension of the PE of the Transformer (Vaswani et al., 2017). 1 Throughout this paper, input means the input of an encoder or a decoder rather than the input of the NMT model (i.e., only source sentences), and output is similar. The NSD was firstly proposed by Shen et al. (2018). This is the first method of linearizing a constituent tree with a sequence of length n , without loss of information, where n is the number of words in the sentence. Formally, given the sentence w = ( w 1 , . . . , w n ) , for any pairs of contiguous words ( w i , w i +1 ) , we can define an NSD d ( w i ) , 2 where i  [1 , n  1] . In Shen et al. (2018), the NSD d S ( w i ) is defined as the height of the lowest common ancestor (LCA) of the words. 3 Subsequently, in Gomez-Rodrguez and Vilares (2018), the NSD d G ( w i ) was defined as the number of the common ancestors of the words. To make the definition complete, we define d ( w n ) as follows: 4 d S ( w n ) = H, d G ( w n ) = 0 , (1) where H is the height of the constituent tree. It is easy to prove that d S ( w i ) + d G ( w i ) = H, i  [1 , n ] . We call d S and d G the absolute NSD. Furthermore, Gomez-Rodrguez and Vilares (2018) define the relative NSD as follows: d R ( w i ) = (cid:40) d G ( w 1 ) , i = 1 , d G ( w i )  d G ( w i  1 ) , i  [2 , n ] . Figure 1 illustrates these NSDs. It is easy to see the one-to-one correspondence relationship between the constituent tree and the (absolute or relative) NSDs. The effectiveness of all different NSDs has been proven on constituent parsing. However, there has been no attempt to use NSD in machine translation. 4 d S ( w n ) and d G ( w n ) are undefined in both of the original papers. We give the definitions here to enable the use of NSD in NMT later. There are many previous studies on using dependency trees to improve NMT (Nguyen Le et al., 2017; Wu et al., 2017). Therefore, we extend NSD to dependency trees. Formally, the dependency NSD between two nodes is defined as follows: d D ( w i ) = i  h ( i ) , (4) where h ( i ) is the index of the head of w i , and we let the index of root be 0 . Note that d D ( w i ) can be either positive or negative, representing the directional information. Figure 2 gives an example. It is easy to see that for w = ( w 1 , . . . , w n ) , the lengths of d S , d G , d R and d D are all n . Denoting the NSD sequence as d = ( d 1 , . . . , d n ) , we can see that d i  Z , i  [1 , n ] , so we can obtain a sequence of embedding vectors e d = ( e d 1 , . . . , e dn ) as follows: e di = E d [ d di + (max( d )  min( d ) + 1)] . We call E d the distance embedding matrix and call e d the syntactic embedding sequence. Note that d can be the NSD on either the source side or the target side, so there are two possible E d , which are denoted as E sd and E td , respectively. The embeddings are calculated as follows: x si = f emb ( E sw [ w si ] , e dsi ) , (6) x ti = f emb ( E tw [ w ti ] , e dti ) , (7) where e dsi and e dti are defined in Eq. 5 on the source side and target side, respectively, and E sw and E tw are the word embedding matrices on both sides, respectively. Inspired by Sennrich and Had-dow (2016), function f emb is used to combine two vectors. This function has many different options, such as: f (cid:107) emb ( x, e ) = x (cid:107) e, (8) f + emb ( x, e ) = x + e, (9) f Wbemb ( x, e ) = W f  ( x (cid:107) e ) + b f , (10) where x, e, b f  R d and W f  R d  2 d . The operator  (cid:107)  is the concatenation of two vectors. When NSD is used as the input sequence on the target side, there is one problem: e dt is unknown during testing. For this case, we use NSDs for both the input and output sequences, let the decoder predict NSD on-the-fly using the strategy introduced in Section 3.3, and use the predicted NSD to calculate e dt . An NSD can be used to form the output sequence to improve NMT using the idea of multi-task learning. Specifically, we train the model to predict the NSD sequence. When NSD is used as the output sequence of the encoder , we minimize the distance (e.g., cross entropy L entdist , see Section 3.5 for details) between the predicted and the golden NSD sequences. When NSD is used as the output sequence of the decoder , besides minimizing the distance, we use the predicted NSD as the input of the next time step. Denote the hidden vector as h = ( h 1 , . . . , h n ) . For the encoder, h i = h si and n = n s , while for the decoder, h i = h ti and n is the current time step of decoding. Then, we can obtain a sequence of predicted syntactic distance  d = (  d 1 , . . . ,  d n ) , which is calculated as follows: p (  d i | h i ) = softmax( W d  h i + b d ) , (11) where W d and b d are parameters to be learned. By minimizing the distance between  d i and d i , NSD can be used to enhance NMT. PE is used by the Transformer (Vaswani et al., 2017) to encode the positions of words. Formally, it is defined as follows: x (cid:48) i = x i + P E ( i ) , (12) P E ( i ) 2 k = sin( i/ 10000 2 k/d ) , (13) P E ( i ) 2 k +1 = cos( i/ 10000 2 k/d ) , (14) where x i can be either x si or x ti , and d is the dimension of the embedding vector. Similarly, we define syntactic PE as follows: P E ( i ) 2 k = sin (cid:16) i + max( d )  min( d )  SPE 2 k/d  2  (cid:17) , (15) P E ( i ) 2 k +1 = cos (cid:16) i + max( d )  min( d )  SPE 2 k/d  2  (cid:17) , (16) where  SPE is a hyperparameter to be tuned. In this way, the periods of these two functions vary from 1 to  SPE . We define syntactic PE in this way because (1) according to a quantitative analysis of the experimental datasets, we found that the ranges of possible values are quite different between NSD and word positions, so we tuned  SPE instead of fixed it to 10000 as in Eqs. 13 and 14, and (2) d i may be negative, so we adjust it to be positive. Instead of using conventional cross-entropy loss function during training, we use the following loss function to make the NMT model learn NSD better: entdist L = LNMT + L dist + L . (17) The first item is the cross-entropy loss of the NMT model, which is LNMT =  (cid:88) (cid:104) w s , w t (cid:105)D log p ( w t | w s ) , (18) where D is the training dataset. The second item is the distance-aware loss, which is inspired by Shen et al. (2018) and is as follows: L dist = (cid:88) (cid:104) w s , w t (cid:105)D ( L sdist ( w s ) + L tdist ( w t )) , L sdist ( w s ) = n s (cid:88) i =1 ( d i   d i ) 2 + (cid:88) i,j>i [1  sign( d i  d j )(  d i   d j )] + , (19) and L tdist can be defined similarly. The third item is the cross-entropy loss for NSD, which is as follows: L entdist = (cid:88) (cid:104) w s , w t (cid:105)D ( L ent ( s ) dist ( w s ) + L ent ( t ) dist ( w t )) , L ent ( s ) dist ( w s ) =  (cid:88) d i  d s p ( d i | h i ) log p (  d i | h i ) , (20) and L ent ( t ) dist can be defined similarly. We experimented on two corpora: (1) ASPEC (Nakazawa et al., 2016), using the top 100 K sentence pairs for training EnJa models and top 1 M sentence pairs for training JaEn models, and (2) LDC, 5 which contains about 1.2M sentence pairs, for training EnCh and ChEn models. To tackling the problem of memory consumption, sentences longer than 150 were filtered out, so that models can be trained successfully. Chinese sentences were segmented by the Stanford segmentation tool. 6 For Japanese sentences, we followed the preprocessing steps recommended in WAT 2017. 7 The test set is a concatenation of NIST MT 2003, 2004, and 2005. Constituent trees are generated by the parser of Kitaev and Klein (2018) 8 , and dependency trees are generated by the parser of Dyer et al. (2015) 9 . Note that although we only used syntactic information of English in our experiments, our method is also applicable to other languages. We implemented our method on OpenNMT 10 (Klein et al., 2017), and used the Transformer as our baseline. As far as we know, there are no previous studies on using syntactic informations in the Transformer. The vocabulary sizes for all languages are 50 , 000 . Both the encoder and decoder have 6 layers. The dimensions of hidden vectors and word embeddings are 512 . The multi-head attention has 5 LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08, and LDC2005T06. 6 https://nlp.stanford.edu/software/ stanford-segmenter-2017-06-09.zip 7 http://lotus.kuee.kyoto-u.ac.jp/WAT/ WAT2017/baseline/dataPreparationJE.html 8 https://github.com/nikitakit/ self-attentive-parser 9 https://github.com/clab/lstm-parser 10 http://opennmt.net 8 heads, and the dropout probability is 0 . 1 (Srivas-tava et al., 2014). The number of training epochs was fixed to 50 , and we used the model which performs the best on the development set for testing. As for optimization, we used the Adam optimizer (Kingma and Ba, 2014), with  1 = 0 . 9 ,  2 = 0 . 998 , and (cid:15) = 10  9 . Warmup and decay strategy for learning rate of Vaswani et al. (2017) are also used, with 8 , 000 warmup steps. We also used the label smoothing strategy (Szegedy et al., 2016) with (cid:15) ls = 0 . 1 . Table 1 compares the effects of the strategies. We evaluate the proposed strategies using character-level BLEU (Papineni et al., 2002) for Chinese and Japanese, and case-insensitive BLEU for English. Comparison of different NSDs. The first five rows of Table 1 compare the results of using different NSDs. When NSD was used at the source side (EnJa/EnCh), all kinds of NSDs improved translation performance. This indicates that NSD can be regarded as a useful linguistic feature to improve NMT. In contrast, when NSD was used at the target side (JaEn/ChEn), d S and d G hurt the performance. This is because the values of d S and d G are volatile. A tiny change of syntactic structure often causes a big change of d S and d G . Since the model has to predict the NSD during decoding, once there is one error, the subsequent predictions will be heavily influenced. The use of d R and d D remedies this problem. Furthermore, the effects of d S and d G are similar, because they are equivalent in nature (refer to Eq. 2). NSD as PE. Rows 5 to 8 of Table 1 evaluate the use of dependency NSD ( d D ) as syntactic PE. Note that for all the experiments, we used not only the syntactic PE but the conventional PE. Experiment results show that this strategy is indeed useful. When the dominators of Eqs. 15 and 16,  SPE , were set to 10 4 , there was no improvement. When they were set to 40 , the improvement was remarkable. This indicates that our design of syntactic PE is reasonable. NSD as input/output and source/target sequences. Rows 8 to 12 of Table 1 are the results of using dependency NSD (i.e., d D ) as the input and/or output sequences on both sides. First, for the choice of f emb , we can see that f (cid:107) emb and f + emb are similar, while f Wbemb yields better performance. This is because the model has to learn W f and Type I/O SPE Loss EnJa JaEn EnCh ChEn 1 N/A N/A No Eq. b f , which increases the model capacity. Second, performance improved for using NSDs both as input and output sequences, and combining both obtained further improvement. Third, NSDs improved the performance both on the source and the target sides. All these results indicate the robustness of NSDs. Effects of distance-aware training. The last three rows compare the different effects of the items in the loss function. When only LNMT are used, the performance is extremely poor. This is within expectations, because with only LNMT , weights related to NSDs are kept to the initial values and were not updated, and hence detrimental to learning. Adding L entdist improves the results sig-nificantly, but the improvement is lower than that of L dist . This is because training with L entdist treats different values of NSDs equally, while L dist penalizes larger differences between the predicted NSD and the golden NSD more severely. We proposed five strategies to improve NMT with NSD. We found relative NSDs and dependency NSDs are able to improve the performance consistently, while absolute NSDs hurt the performance for some cases. The improvement obtained by using NSDs is general in that NSDs can be used at both the source side and target side, both as input sequences and output sequences. Using NSDs as syntactic PE is also useful, and training with a distance-aware loss function is quite important. We are grateful to the anonymous reviewers for their insightful comments and suggestions. Akihiro Tamura is supported by JSPS KAK-ENHI Grant Number JP18K18110. Tiejun Zhao is supported by the National Key Research and Development Program of China via grant 2017YFB1002102.
abstain abstain method abstain objective abstain abstain abstain other abstain abstain abstain objective abstain abstain abstain objective abstain objective objective objective objective abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other method abstain abstain abstain abstain other abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain other method abstain abstain abstain method abstain abstain abstain other other method abstain abstain method abstain abstain method method method abstain abstain abstain abstain other abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain abstain other other other