Position encoding (PE), an essential part of self-attention networks (SANs), is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. However, in cross-lingual scenarios, e.g., machine translation, the PEs of source and target sentences are modeled independently. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. In this paper, we augment SANs with cross-lingual position representations to model the bilingually aware latent structure for the input sentence. Specifically, we utilize bracketing transduction grammar (BTG)-based reordering information to encourage SANs to learn bilingual diagonal alignments. Experimental results on WMT'14 English  German, WAT'17 Japanese  English, and WMT'17 Chinese  English translation tasks demonstrate that our approach significantly and consistently improves translation quality over strong baselines. Extensive analyses confirm that the performance gains come from the cross-lingual information. Although self-attention networks (SANs) (Lin et al., 2017) have achieved the state-of-the-art performance on several natural language processing (NLP) tasks (Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2018), they possess the innate disadvantage of sequential modeling due to the lack of positional information. Therefore, absolute position encoding (APE) (Vaswani et al., 2017) and relative position encoding (RPE) (Shaw et al., 2018) were introduced to better capture the sequential dependencies. However, either absolute or relative PE is language-independent and its embedding Bush with Sharon held a talk Bush held a talk with Sharon [ source ] [ re-ordered ] 0 1 2 3 4 5 [ abs POS] 0 3 4 5 1 2 [ XL POS]         [ target ]         Bush held a talk with Sharon remains fixed. This inhibits the capacity of SANs when modelling multiple languages, which have diverse word orders and structures (Gell-Mann and Ruhlen, 2011). Recent work have shown that modeling cross-lingual information ( e.g., alignment or reordering) at encoder or attention level improves translation performance for different language pairs (Cohn et al., 2016; Du and Way, 2017; Zhao et al., 2018; Kawara et al., 2018). Inspired by their work, we propose to augment SANs with cross-lingual representations , by encoding reordering indices at embedding level. Taking English  Chinese translation task for example, we first reorder the English sentence by deriving a latent bracketing transduction grammar (BTG) tree (Wu, 1997) (Fig. 1a). Similar to absolute position, the reordering information can be represented as cross-lingual position (Fig. 1b). In addition, we propose two strategies to incorporate cross-lingual position encoding into SANs. We conducted experiments on three commonly-cited datasets of machine translation. Results show that exploiting cross-lingual PE consistently improves translation quality . Further analysis reveals that our method improves the alignment quality (Sec. 4.3) and context-free Transformer (Tang et al., 2019) (Sec. 4.4). Furthermore, contrastive evaluation demonstrates that NMT models benefits from the cross-lingual information rather than denoising ability (Sec. 4.5). To tackle the position unaware problem, absolute position information is injected into the SANs: where pos abs denotes the numerical position indices, i is the dimension of the position indices and d model means hidden size. f (  ) alternately employs sin (  ) and cos (  ) for even and odd dimensions. Accordingly, the position matrix PE can be obtained given the input X = { x 1 , . . . , x T }  RT  d model . Then, the position aware output Z is calculated by: Z = X + PE abs  RT  d model (2) Self-Attention The SANs compute the attention of each pair of elements in parallel. It first converts the input into three matrices Q , K , V , representing queries, keys, and values, respectively: { Q , K , V } = { ZWQ , ZWK , ZWV } (3) where WQ , WK , WV  R d model  d model are parameter matrices. The output is then computed as a weighted sum of values by ATT ( Q , K , V ) . SANs can be implemented with multi-head attention mechanism, which requires extra splitting and concatenation operations. Specifically, WQ , WK , WV and Q , K , V in Eq. (3) is split into H sub-matrices, yielding H heads. For the h -th head, the output is computed by: O h = ATT ( Q h , K h , V h )  RT  d v (4) Where subspace parameters are W hQ , W hK  R d model  d k and W hV  R d model  d v , where d k , d v + Nonlinear Fusion refer to the dimensions of keys and values in the subspace, and normally d k = d v = d model / H. Finally, these subspaces are combined with concatenation operation: O = CONCAT ( O 1 , . . . , OH ) WO (5) where WO  R Hd v  d model and O  RT  d model are the parameter matrix and output, respectively. First, we built a BTG-based reordering model (Neu-big et al., 2012) to generate a reordered source sentence according to the word order of its corresponding target sentence. Second, we obtained the reordered word indices pos XL that correspond with the input sentence X . To output the cross-lingual position matrix PEXL , we inherit the sinusoidal function in Eq. (1). Formally, the process is: PEXL = f ( BTG ( X )) (6) 3.2 Integration Strategy As shown in Fig. 2, we propose two strategies to integrate the cross-lingual position encoding (XLPE) into SANs: inputting-level XL ( InXL ) SANs and head-level ( HeadXL ) SANs. Inputting-level XL SANs As illustrated in Fig. 2a, we employ a non-linear function TANH (  ) to fuse PE abs and PEXL : PEIN-XL = TANH ( PE abs U + PEXLV ) (7) where U , V are trainable parameters. In our preliminary experiments, the non-linear function performs better than element-wise addition. This might because complex non-linear one have better fitting capabilities, thereby avoiding exceptional reordering to some extent. Next, we perform Eq. (2) to obtain the output representations: ZIN-XL = X + PEIN-XL (8) Similarly, we use Eq. (3)  (5) to calculate multiple heads of SANs. Head-level XL SANs Instead of projecting XL PE to all attention heads, we feed partial of them, such that some heads contain XL PE and others contain APE, namely HeadXL. As shown in Fig. 2b, we fist add APE and XL PE for X , respectively: Z abs = X + PE abs ZXL = X + PEXL (9) We denote the number of XL PE equipped heads as   { 0 , . . . , H } . To perform the attention calculation, W i is divided into [ WXL i  R d model  d v ; W absi  R d model  ( H   ) d v ] for each i  Q , K , V , correspondingly generating two types of { Q , K , V } for XL PE heads and APE heads. According to Eq. (4), the output of each XL PE head is: OXL h = ATT ( QXL h , KXL h , VXL h )  RT  d v (10) As a result, the final output of HeadXL is: HEADSAN ( X ) = CONCAT ( OXL 1 , . . . , OXL  O abs +1 , . . . , O absH ) WO (11) In particular,  = 0 refers to the original Transformer (Vaswani et al., 2017) and  = H means that XL PE will propagate over all attention heads. We conduct experiments on word order-diverse language pairs: WMT'14 English  German (En-De), WAT'17 Japanese  English (Ja-En), and WMT'17 Chinese  English (Zh-En & En-Zh). For English  German, the training set consists of 4.5 million sentence pairs and newstest2013 & 2014 are used as the dev. and test sets, respectively. BPE with 32K merge operations is used to handle low-frequency words. For Japanese  English, we follow Morishita et al. (2017) to use the first two sections as training data, which consists of 2.0 million sentence pairs. The dev. and test sets contain 1790 and 1812 sentences. For Chinese  English, we follow Hassan et al. (2018) to get 20 million 28.3 28.6 28.8 0 2 4 6 8 10 12 14 BLEU  (#heads with XL PE) Transformer Big Head XL SANs Figure 3: BLEU score on newstest2014 for different  . sentence pairs. We develop on devtest2017 and test on newstest2017. We use SacreBLEU (Post, 2018) as the evaluation metric with statistical significance test (Collins et al., 2005). We evaluate the proposed XL PE strategies on Transformer. The baseline systems include Relative PE (Shaw et al., 2018) and directional SAN (DiSAN, Shen et al. 2018). We implement them on top of OpenNMT (Klein et al., 2017). In addition, we report the results of previous studies (Hao et al., 2019; Wang et al., 2019; Chen et al., 2019b,a; Du and Way, 2017; Hassan et al., 2018). The reordered source sentences are generated by BTG-based preordering model (Neubig et al., 2012) trained with above sub-word level 1 parallel corpus. At training phase, we first obtain word alignments from parallel data using GIZA++ or FastAlign, and then the training process is to find the optimal BTG tree for source sentence consistent with the order of the target sentence based on the word alignments and parallel data. At decoding phase, we only provide source sentences as input and the model can output reordering indices, which will be fed into NMT model. Thus, bilingual alignment information is only used to preprocess training data, but not necessary at decoding time. For fair comparison, we keep the Transformer decoder unchanged and validate different position representation strategies on the encoder. We conduct all experiments on the TRANSFORMER-BIG with four V100 GPUs. Fig. 3 reports the results of different  for Head XL SAN s. With increasing of XL PE-informed heads, the best BLEU is achieved when #heads = 4, which is therefore left as the default setting for HeadXL. Then, the BLEU score gradually decreases as the 1 Garg et al. (2019) show that sub-word units are beneficial for statistical model. number of APE-informed heads decrease (   ), indicating that sequential position embedding is still essential for SANs. Tab. 1 shows the results on En-De, inputting-level cross-lingual PE (+InXL PE) and head-level cross-lingual PE (+HeadXL PE) outperform Transformer BIG by 0.30 and 0.36 BLEU points, and combining these two strategies 2 achieves a 0.69 BLEU point increase. For Ja-En, Zh-En, and En-Zh (Tab. 2), we observe a similar phenomenon, demonstrating that XL PE on SANs do improve the translation performance for several language pairs. It is worth noting that our approach introduces nearly no additional parameters (+0.01M over 282.55M). Our proposed XL PE intuitively encourages SANs to learn bilingual diagonal alignment, so has the 2 Replace PEXL in Eq. (9) with PEIN-XL in Eq. (8). potential to induce better attention matrices. We explore this hypothesis on the widely used Gold Alignment dataset 3 and follow Tang et al. (2019) to perform the alignment. The only difference being that we average the attention matrices across all heads from the penultimate layer (Garg et al., 2019). The alignment error rate (AER, Och and Ney 2003), precision (P) and recall (R) are reported as the evaluation metrics. Tab. 3 summarizes the results. We can see: 1) XL PE allows SANs to learn better attention matrices, thereby improving alignment performance (27.4 / 26.9 vs. 29.7); and 2) combining the two strategies delivers consistent improvements (24.7 vs. 29.7). Tang et al. (2019) showed that context-free Transformer (directly propagating the source word em-beddings with PE to the decoder) achieved comparable results to the best RNN-based model. We argue that XL PE could further enhance the context-free Transformer. On English  German dataset, 3 http://www-i6.informatik.rwth-aachen. de/goldAlignment , the original dataset is German-English, we reverse it to English-German. we compare LSTM-based model, Transformer BIG -noenc-nopos, +APE, +RPE and +InXL PE. For fair comparison, we set the LSTM hidden size to 1024. In Tab. 4, we can see: 1) position information is the most important component for the context-free model, bringing +14.45 average improvement; 2) InXL PE equipped context-free Transformer significantly outperforms the LSTM model while consuming less parameters; and 3) compared to the increment on standard Transformer (+0.30 over 28.36), InXL PE improves more for context-free Transformer (+0.57 over 24.11), where the improvements are +2.3% vs. +1.1%. To demonstrate that our improvements come from cross-lingual position information rather than noisy position signals, we attack our model by adding noises 4 into reordered indices of training sentences. As shown in Fig. 4, our method can tolerate partial reordering noises and maintain performance to some extent. However, as noise increases, translation quality deteriorates, indicating that noises in reordering information do not work as regularization. This contrastive evaluation also confirms that the model does not benefit from the noise as much as it benefits from the reordering information. Augmenting SANs with position representation SANs ignore the position of each token due to its position-unaware bag-of-words assumption. The most straightforward strategy is adding the position representations as part of the token representations (Vaswani et al., 2017; Shaw et al., 2018). Besides above sequential PE approaches, Wang et al. (2019) enhanced SANs with structural positions extracted from the syntax dependencies. However, none of them considered modeling the cross-4 We randomly swap two reordered positional indexes with different ratios. Modeling cross-lingual divergence There has been many works modeling cross-lingual divergence ( e.g. , reordering) in statistical machine translation (Nagata et al., 2006; Durrani et al., 2011, 2013). However, it is difficult to migrant them to neural machine translation. Kawara et al. (2018) pre-reordered the source sentences with a recursive neural network model. Chen et al. (2019a) learned the reordering embedding by considering the relationship between the position embedding of a word and SANS -calculated sentence representation. Yang et al. (2019) showed that SANs in machine translation could learn word order mainly due to the PE, indicating that modeling cross-lingual information at position representation level may be informative. Thus, we propose a novel cross-lingual PE method to improve SANs. In this paper, we presented a novel cross-lingual position encoding to augment SANs by considering cross-lingual information ( i.e., reordering indices) for the input sentence. We designed two strategies to integrate it into SANs. Experiments indicated that the proposed strategies consistently improve the translation performance. In the future, we plan to extend the cross-lingual position encoding to non-autoregressive MT (Gu et al., 2018) and unsupervised NMT (Lample et al., 2018). This work was supported by Australian Research Council Projects FL-170100117. We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions.
abstain abstain abstain objective method objective abstain abstain abstain abstain abstain abstain abstain objective objective abstain objective method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method method abstain method method abstain method method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain result abstain result abstain abstain other other other method abstain abstain abstain abstain abstain abstain objective method abstain abstain other other