Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this problem, we propose regressing word embeddings (ReWE) as a new regularization technique in a system that is jointly trained to predict the next word in the translation (categorical value) and its word embedding (continuous value). Such a joint training allows the proposed system to learn the distributional properties represented by the word embeddings, empirically improving the generalization to unseen sentences. Experiments over three translation datasets have showed a consistent improvement over a strong baseline, ranging between 0 . 91 and 2 . 54 BLEU points, and also a marked improvement over a state-of-the-art system. The last few years have witnessed remarkable improvements in the performance of machine translation (MT) systems. These improvements are strongly linked to the development of neural machine translation (NMT): based on encoder-decoder architectures (also known as seq2seq), NMT can use recurrent neural networks (RNNs) (Sutskever et al., 2014; Cho et al., 2014; Wu et al., 2016), convolutional neural networks (CNNs) (Gehring et al., 2017) or transformers (Vaswani et al., 2017) to learn how to map a sentence from the source language to an adequate translation in the target language. In addition, attention mechanisms (Bahdanau et al., 2015; Luong et al., 2015) help soft-align the encoded source words with the predictions, further improving the translation. pointed out by (Elbayad et al., 2018), MLE suffers from two obvious limitations: the first is that it treats all the predictions other than the ground truth as equally incorrect. As a consequence, synonyms and semantically-similar words  which are often regarded as highly interchangeable with the ground truth  are completely ignored during training. The second limitation is that MLE-trained systems suffer from exposure bias (Ben-gio et al., 2015; Ranzato et al., 2015) and do not generalize well over the large output space of translations. Owing to these limitations, NMT systems still struggle to outperform other traditional MT approaches when the amount of supervised data is limited (Koehn and Knowles, 2017). In this paper, we propose a novel regularization technique for NMT aimed to influence model learning with contextual properties. The technique  nicknamed ReWE from regressing word em-bedding  consists of modifying a conventional seq2seq decoder to jointly learn to a) predict the next word in the translation (categorical value), as usual, and b) regress its word embedding (numer-ical value). Figure 1 shows the modified decoder. Both predictions are incorporated in the training objective, combining standard MLE with a continuous loss function based on word embeddings. The rationale is to encourage the system to learn to co-predict the next word together with its context (by means of the word embedding representation), in the hope of achieving improved generalization. At inference time, the system operates as a standard NMT system, retaining the categorical prediction and ignoring the predicted embedding. We qualify our proposal as a regularization technique since, like any other regularizers, it only aims to influence the model's training, while leaving the inference unchanged. We have evaluated the proposed system over three translation datasets of different size, namely English-French (en-fr), Czech-English (cs-en), and Basque-English (eu-en). In each case, ReWE has significantly outperformed its baseline, with a marked improvement of up to 2.54 BLEU points for eu-en, and consistently outperformed a state-of-the-art system (Denkowski and Neubig, 2017). A substantial literature has been devoted to improving the generalization of NMT systems. Fadaee et al. (2017) have proposed a data augmentation approach for low-resource settings that generates synthetic sentence pairs by replacing words in the original training sentences with rare words. Kudo (2018) has trained an NMT model with different subword segmentations to enhance its robustness, achieving consistent improvements over low-resource and out-of-domain settings. Zhang et al. (2018) have presented a novel regularization method that encourages target-bidirectional agreement. Other work has proposed improvements over the use of a single ground truth for training: Ma et al. (2018) have augmented the conventional seq2seq model with a bag-of-words loss under the assumption that the space of correct translations share similar bag-of-words vectors, achieving promising results on a Chinese-English translation dataset; Elbayad et al. (2018) have used sentence-level and token-level reward distributions to smooth the single ground truth. Chousa et al. (2018) have similarly leveraged a token-level smoother. In a recent paper, Denkowski and Neubig (2017) have achieved state-of-the-art translation accuracy by leveraging a variety of techniques which include: dropout (Srivastava et al., 2014), lexicon bias (Arthur et al., 2016), pre-translation (Niehues et al., 2016), data bootstrapping (Chen et al., 2016), byte-pair encoding (Sennrich et al., 2016) and ensembles of independent models (Rokach, 2010). However, to our knowledge none of the mentioned approaches have explicitly attempted to leverage the embeddings of the ground-truth tokens as targets. For this reason, in this paper we explore regressing toward pre-trained word embeddings as an attempt to capture contextual properties and achieve improved model regularization. The model is a standard NMT model with attention in which we use RNNs for the encoder and decoder. Following the notation of (Bahdanau et al., 2015), the RNN in the decoder generates a sequence of hidden vectors, { s 1 , . . . , s m } , given the context vector, the previous hidden state s j  1 and the previous predicted word y j  1 : s j = dec rnn ( s j  1 , y j  1 , c j ) j = 1 , . . . , m (1) where y 0 and s 0 are initializations for the state and label chains. Each hidden vector s j (of parameter size S ) is then linearly transformed into a vector of vocabulary size, V , and a softmax layer converts it into a vector of probabilities (Eq. 2), where W (a matrix of size V  S ) and b (a vector of size V  1 ) are learnable parameters. The predicted conditional probability distribution over the words in the target vocabulary, p j , is given as: p j = softmax ( Ws j + b ) (2) As usual, training attempts to minimize the negative log-likelihood (NLL), defined as: NLL loss =  m (cid:88) j =1 log( p j ( y j )) (3) where p j ( y j ) notes the probability of ground-truth word y j . The NLL loss is minimized when the probability of the ground truth is one and that of all other words is zero, treating all predictions different from the ground truth as equally incorrect. Pre-trained word embeddings (Pennington et al., 2014; Bojanowski et al., 2017; Mikolov et al., 2013) capture the contextual similarities of words, typically by maximizing the probability of word w t + k to occur in the context of center word w t . This probability can be expressed as: p ( w t + k | w t ) ,  c  k  c, k (cid:54) = 0 t = 1 , . . . , T (4) where c is the size of the context and T is the total number of words in the training set. Traditionally, word embeddings have only been used as input representations. In this paper, we instead propose using them in output as part of the training objective, in the hope of achieving regularization and improving prediction accuracy. Building upon the baseline model presented in Section 3.1, we have designed a new joint learning setting: our decoder still predicts the probability distribution over the vocabulary, p j (Eq. 2), while simultaneously regressing the same shared s j to the ground-truth word embedding, e ( y j ) . The ReWE module consists of two linear layers with a Rectified Linear Unit (ReLU) in between, outputting a vector e j of word embedding size (Eq. 5). Please note that adding this extra module adds negligible computational costs and training time. Full details of this module are given in the supplementary material. In the experiment, we have explored two cases for the ReW E loss : the minimum square error (MSE) 1 and the cosine embedding loss (CEL) 2 . Finally, the NLL loss and the ReW E loss are combined to form the training objective using a positive trade-off coefficient,  : Loss = NLL loss + ReW E loss (7) As mentioned in the Introduction, at inference time we ignore the ReWE output, e j , and the model operates as a standard NMT system. 1 https://pytorch.org/docs/stable/nn.html#torch.nn. MSELoss 2 https://pytorch.org/docs/stable/nn.html#torch.nn. CosineEmbeddingLoss Dataset Size Sources IWSLT16 en-fr 219 , 777 TED talks IWSLT16 cs-en 114 , 243 TED talks WMT16 eu-en 89 , 413 IT-domain data Dataset Validation set Test set en-fr TED test 2013+2014 TED test 2015+2016 cs-en TED test 2012+2013 TED test 2015+2016 eu-en Sub-sample of PaCo IT-domain test Table 1: Top: parallel training data. We have developed our models building upon the OpenNMT toolkit (Klein et al., 2017) 3 . For training, we have used the same settings as (Denkowski and Neubig, 2017). We have also explored the use of sub-word units learned with byte pair encoding (BPE) (Sennrich et al., 2016). All the preprocessing steps, hyperparameter values and training parameters are described in detail in the supplementary material to ease reproducibility of our results. We have evaluated these systems over three publicly-available datasets from the 2016 ACL Conference on Machine Translation (WMT16) 4 and the 2016 International Workshop on Spoken Language Translation (IWSLT16) 5 . Table 1 lists the datasets and their main features. Despite having nearly 90,000 parallel sentences, the eu-en dataset only contains 2,000 human-translated sentences; the others are translations of Wikipedia page titles and localization files. Therefore, we regard the eu-en dataset as very low-resource. In addition to the seq2seq baseline, we have compared our results with those recently reported by Denkowski and Neubig for non-ensemble models (2017). For all models, we report the BLEU scores (Papineni et al., 2002), with the addition of selected comparative examples. Two contrastive experiments are also added in supplementary notes. As a preliminary experiment, we have carried out a sensitivity analysis to determine the optimal value of the trade-off coefficient,  (Eq. 6), using the en-fr validation set. The results are shown in Figure 2, where each point is the average of three runs trained with different seeds. The figure shows that 3 Our code can be found at: https://github.com/ijauregiCMCRC/ReWE NMT 4 WMT16: http://www.statmt.org/wmt16/ 5 IWSLT16: https://workshop2016.iwslt.org/ Models en-fr cs-en eu-en Word BPE Word BPE Word BPE (Denkowski and Neubig, 2017) 33.60 34.50 21.00 22.60 (Denkowski and Neubig, 2017) + Dropout 34.5 34.70 21.4 23.60 (Denkowski and Neubig, 2017) + Lexicon 33.9 34.80 20.6 22.70 (Denkowski and Neubig, 2017) + Pre-translation N/A 34.90 N/A 23.80 (Denkowski and Neubig, 2017) + Bootstrapping 34.40 35.20 21.60 23.60 Our baseline 34.16 34.09 20.57 22.69 12.14 17.17 Our baseline + ReWE (CEL) (  = 20 ) 35.52 35.22 21.83 23.60 13.73 19.71 Table 2: BLEU scores over the test sets. the MSE loss has outperformed slightly the baseline for small values of  ( < 1 ), but the BLEU score has dropped drastically for larger values. Conversely, the CEL loss has increased steadily with  , reaching 38 . 23 BLEU points for  = 20 , with a marked improvement of 1 . 53 points over the baseline. This result has been encouraging and therefore for the rest of the experiments we have used CEL as the ReW E loss and kept the value of  to 20 . In Section 4.3, we further discuss the behavior of CEL and MSE. Table 2 reports the results of the main experiment for all datasets. The values of our experiments are for blind runs over the test sets, averaged over 10 independent runs with different seeds. The results show that adding ReWE has significantly improved the baseline in all cases, with an average of 1 . 46 BLEU points. In the case of the eu-en dataset, the improvement has reached 2 . 54 BLEU points. We have also run unpaired t-tests between our baseline and ReWE, and the differences have proved statistically significant ( p -values < 0 . 05 ) in all cases. Using BPE has proved beneficial for the cs-en and eu-en pairs, but not for the en-fr Src : Hautatu Kontrol panela  Programa lehenetsiak , eta aldatu bertan . pair. We speculate that English and French may be closer to each other at word level and, therefore, less likely to benefit from the use of sub-word units. Conversely, Czech and Basque are morphologically very rich, justifying the improvements with BPE. Table 2 also shows that our model has outperformed almost all the state-of-the-art results reported in (Denkowski and Neubig, 2017) (dropout, lexicon bias, pre-translation, and bootstrapping), with the only exception of the pre-translation case for the cs-en pair with BPE. This shows that the proposed model is competitive with contemporary NMT techniques. To further explore the improvements obtained with ReWE, we have qualitatively compared several translations provided by the baseline and the baseline + ReWE (CEL), trained with identical seeds. Overall, we have noted a number of instances where ReWE has provided translations with more information from the source (higher ad-equacy). For reasons of space, we report only one example in Table 3, but more examples are available in the supplementary material. In the example, the baseline has chosen a generic word, pro-gram, while ReWE has been capable of correctly predicting Default Program and being specific about the object, it. To further explore the behaviour of the ReWE loss, Figure 3 plots the values of the NLL and ReWE (CEL) losses during training of our model over the en-fr training set. The natural values of the ReWE (CEL) loss (blue, dashed) are much lower than those of the NLL loss (red, + ), and thus its contribution to the gradient is likely to be limited. However, when scaled up by a factor of  = 20 (magenta,  ), its influence on the gradient becomes more marked. Empirically, both the NLL and ReWE (CEL) losses decrease as the training progresses and the total loss (green,  ) decreases. As shown in the results, this combined training objective has been able to lead to improved translation results. Conversely, the MSE loss has not exhibited a similarly smooth behaviour (supplementary mate-rial). Even when brought to scale with the NLL loss, it shows much larger fluctuations as the training progresses. In particular, it shows major increases at the re-starts of the optimizer for the simulated annealing that are not compensated for by the rest of the training. It is easy to speculate that the MSE loss is much more sensitive than the cosine distance to the changes in the weights caused by dropout and the re-starts. As such, it seems less suited for use as training objective. In this paper, we have proposed a new regularization technique for NMT (ReWE) based on a joint learning setting in which a seq2seq model simultaneously learns to a) predict the next word in the translation and b) regress toward its word embedding. The results over three parallel corpora have shown that ReWE has consistently improved over both its baseline and recent state-of-the-art results from the literature. As future work, we plan to extend our experiments to better understand the potential of the proposed regularizer, in particular for unsupervised NMT (Artetxe et al., 2018; Lample et al., 2018). We would like to acknowledge the financial support received from the Capital Markets Cooperative Research Centre (CMCRC), an industry-led research initiative of the Australian Government. We would also like to thank Ben Hachey, Michael Nolan and Nadia Shnier for their careful reading of our paper and their insightful comments. Finally, we are grateful to the anonymous reviewers for all their comments and suggestions.
abstain objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective objective abstain other other other other other other other abstain objective method other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain other other other