We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a con-stant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to + 4.3 BLEU, surpassing phrase-based translation in nearly all settings. 1 1 Introduction Neural network approaches to machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015a; Gehring et al., 2017) are appealing for their single-model, end-to-end training process, and have demonstrated competitive performance compared to earlier statistical approaches (Koehn et al., 2007; Junczys-Dowmunt et al., 2016). However, there are still many open problems in NMT (Koehn and Knowles, 2017). One particular issue is mistranslation of rare words. For example, consider the Uzbek sentence: Source: Ammo muammolar hali ko'p, deydi amerikalik olim Entoni Fauchi. Reference: But still there are many problems, says American scientist Anthony Fauci. Baseline NMT: But there is still a lot of problems, says James Chan. At the position where the output should be Fauci , the NMT model's top three candidates are Chan , 1 The code for this work can be found at https://github.com/tnq177/improving_lexical_choice_in_nmt Fauci , and Jenner . All three surnames occur in the training data with reference to immunologists: Fauci is the director of the National Institute of Allergy and Infectious Diseases, Margaret (not James) Chan is the former director of the World Health Organization, and Edward Jenner invented smallpox vaccine. But Chan is more frequent in the training data than Fauci , and James is more frequent than either Anthony or Margaret . Because NMT learns word representations in continuous space, it tends to translate words that seem natural in the context, but do not reflect the content of the source sentence (Arthur et al., 2016). This coincides with other observations that NMT's translations are often fluent but lack accuracy (Wang et al., 2017b; Wu et al., 2016). (cid:16) (cid:17) where W e and b e are a vector and a scalar depending only on e , and  h is a vector depending only on the source sentence and previous output words. We propose two modifications to this layer. First, we argue that the term W e   h , which measures how well e fits into the context  h , favors common words disproportionately, and show that it helps to fix the norm of both vectors to a constant. Second, we add a new term representing a more direct connection from the source sentence, which allows the model to better memorize translations of rare words. Below, we describe our models in more detail. Then we evaluate our approaches on eight language pairs, with training data sizes ranging from 100k words to 8M words, and show improvements of up to + 4.3 BLEU, surpassing phrase-based translation in nearly all settings. Finally, we provide some analysis to better understand why our modifications work well. Given a source sequence f = f 1 f 2    f m , the goal of NMT is to find the target sequence e = e 1 e 2    e n that maximizes the objective function: X We use the global attentional model with general scoring function and input feeding by Luong et al. (2015a). We provide only a very brief overview of this model here. It has an encoder, an attention, and a decoder. The encoder converts the words of the source sentence into word embeddings , then into a sequence of hidden states . The decoder generates the target sentence word by word with the help of the attention. At each time step t , the attention calculates a set of attention weights a t ( s ). These attention weights are used to form a weighted average of the encoder hidden states to form a context vector c t . From c t and the hidden state of the decoder are computed the attentional hidden state  h t . Finally, the predicted probability distribution of the t 'th target word is: p ( e t | e < t , f ) = softmax( W o  h t + b o ) . 1 The rows of the output layer's weight matrix W o can be thought of as embeddings of the output vocabulary, and sometimes are in fact tied to the embeddings in the input layer, reducing model size while often achieving similar performance (Inan et al., 2017; Press and Wolf, 2017). We verified this claim on some language pairs and found out that this approach usually performs better than without tying, as seen in Table 1. For this reason, we always tie the target embeddings and W o in all of our models. The output word distribution (1) can be written as: (cid:16) (cid:17) where W e is the embedding of e , b e is the e 'th component of the bias b o , and  W e ,  h is the angle between W e and  h . We can intuitively interpret the terms as follows. The term k  h k has the e  ect of sharpening or flattening the distribution, reflect-ing whether the model is more or less certain in a particular context. The cosine similarity cos  W e ,  h measures how well e fits into the context. The bias b e controls how much the word e is generated; it is analogous to the language model in a log-linear translation model (Och and Ney, 2002). Finally, k W e k also controls how much e is generated. Figure 1 shows that it generally correlates with frequency. But because it is multiplied by cos  W e ,  h , it has a stronger e  ect on words whose embeddings have direction similar to  h , and less e  ect or even a negative e  ect on words in other directions. We hypothesize that the result is that the model learns k W e k that are disproportionately large. Observe that cos  W e ,  h and even b e both favor the correct output word Fauci , whereas k W e k favors the more frequent, but incorrect, word Chan . The most frequently-mentioned immunologist trumps other immunologists. To solve this issue, we propose to fix the norm of all target word embeddings to some value r . Followingthe weight normalization approach of Salimans and Kingma (2016), we reparameterize W e as r v e k v e k , but keep r fixed. A similar argument could be made for k  h t k : because a large k  h t k sharpens the distribution, causing frequent words to more strongly dominate rare words, we might want to limit it as well. We compared both approaches on a development set and found that replacing  h t in equation (1) with r  h t k  h t k indeed performs better, as shown in Table 1. 335 2 4 6 8 k W e k ha-entu-en hu-en 0 5 , 000 10 , 000 15 , 000  2  1 0 frequency rank of e b e ha-entu-en hu-en Figure 1: The word embedding norm k W e k generally correlates with the frequency of e , except for the most frequent words. The attentional hidden state  h contains information not only about the source word(s) corresponding to the current target word, but also the contexts of those source words and the preceding context of the target word. This could make the model prone to generate a target word that fits the context but doesn't necessarily correspond to the source word(s). Count-based statistical models, by contrast, don't have this problem, because they simply don't model any of this context. Arthur et al. (2016) try to alleviate this issue by integrating a count-based lexicon into an NMT system. However, this lexicon must be trained separately using GIZA ++ (Och and Ney, 2003), and its parameters form a large, sparse array, which can be di  cult to store in GPU memory. We propose instead to use a simple feedforward neural network (FFNN) that is trained jointly with the rest of the NMT model to generate a target word based directly on the source word(s). Let f s ( s = 1 , . . . , m ) be the embeddings of the source words. We use the attention weights to form a tokens vocab layers  10 6  10 3 num / size ta-en 0.2 / 0.1 4.0 / 3.4 1 / 512 ur-en 0.2 / 0.2 4.2 / 4.2 1 / 512 ha-en 0.8 / 0.8 10.6 / 10.4 2 / 512 tu-en 0.8 / 1.1 21.1 / 13.3 2 / 512 uz-en 1.5 / 1.9 29.8 / 17.4 2 / 512 hu-en 2.0 / 2.3 27.3 / 15.7 2 / 512 en-vi 2.1 / 2.6 17.0 / 7.7 2 / 512 en-ja (BTEC) 3.6 / 5.0 17.8 / 21.8 4 / 768 en-ja (KFTT) 7.8 / 8.0 48.2 / 49.1 4 / 768 Table 2: Statistics of data and models: e  ective number of training source / target tokens, source / target vocabulary sizes, number of hidden layers and number of units per layer. weighted average of the embeddings (not the hidden states, as in the main model) to give an average source-word embedding at each decoding time step t : f  t = tanh X s a t ( s ) f s . Then we use a one-hidden-layer FFNN with skip connections (He et al., 2016): h  t = tanh( W f  t ) + f  t and combine its output with the decoder output to get the predictive distribution over output words at time step t : p ( y t | y < t , x ) = softmax( W o  h t + b o + W  h  t + b  ) . For the same reasons that were given in Section 3 for normalizing  h t and the rows of W ot , we normalize h  t and the rows of W  as well. Note, however, that we do not tie the rows of W  with the word embeddings; in preliminary experiments, we found this to yield worse results. 5 Experiments We conducted experiments testing our normalization approach and our lexical model on eight language pairs using training data sets of various sizes. This section describes the systems tested and our results. 5.1 Data We evaluated our approaches on various language pairs and datasets: 336  Tamil (ta), Urdu (ur), Hausa (ha), Turkish (tu), and Hungarian (hu) to English (en), using data from the LORELEI program.  English to Vietnamese (vi), using data from the IWSLT 2015 shared task. 2  To compare our approach with that of Arthur et al. (2016), we also ran on their English to Japanese (ja) KFTT and BTEC datasets. 3 We tokenized the LORELEI datasets using the default Moses tokenizer, except for Urdu-English, where the Urdu side happened to be tokenized using Morfessor FlatCat ( w = 0 . 5). We used the preprocessed English-Vietnamese and English-Japanese datasets as distributed by Luong et al., and Arthur et al., respectively. Statistics about our data sets are shown in Table 2. 5.2 Systems We compared our approaches against two baseline NMT systems: untied , which does not tie the rows of W o to the target word embeddings, and tied , which does. In addition, we compared against two other baseline systems: Moses : The Moses phrase-based translation system (Koehn et al., 2007), trained on the same data as the NMT systems, with the same maximum sentence length of 50. No additional data was used for training the language model. Unlike the NMT systems, Moses used the full vocabulary from the training data; unknown words were copied to the target sentence. Arthur: Our reimplementation of the discrete lexicon approach of Arthur et al. (2016). We only tried their auto lexicon, using GIZA ++ (Och and Ney, 2003), integrated using their bias approach. Note that we also tied embedding as we found it also helped in this case. Against these baselines, we compared our new systems: fixnorm : The normalization approach described in Section 3. fixnorm + lex : The same, with the addition of the lexical translation module from Section 4. 2 https://nlp.stanford.edu/projects/nmt/ 3 http://isw3.naist.jp/~philip-a/emnlp2016/ 5.3 Details Model For all NMT systems, we fed the source sentences to the encoder in reverse order during both training and testing, following Luong et al. (2015a). Information about the number and size of hidden layers is shown in Table 2. The word embedding size is always equal to the hidden layer size. Following common practice, we only trained on sentences of 50 tokens or less. We limited the vocabulary to word types that appear no less than 5 times in the training data and map the rest to UNK . For the English-Japanese and English-Vietnamese datasets, we used the vocabulary sizes reported in their respective papers (Arthur et al., 2016; Luong and Manning, 2015). For fixnorm , we tried r  { 3 , 5 , 7 } and selected the best value based on the development set performance, which was r = 5 except for English-Japanese (BTEC), where r = 7. For fixnorm + lex , because W s  h t + W  h  t takes on values in [  2 r 2 , 2 r 2 ], we reduced our candidate r values by roughly a factor of  2, to r  { 2 , 3 . 5 , 5 } . A radius r = 3 . 5 seemed to work the best for all language pairs. Training We trained all NMT systems with Adadelta (Zeiler, 2012). All parameters were initialized uniformly from [  0 . 01 , 0 . 01]. When a gra-dient's norm exceeded 5, we normalized it to 5. We also used dropout on non-recurrent connections only (Zaremba et al., 2014), with probability 0.2. We used minibatches of size 32. We trained for 50 epochs, validating on the development set after every epoch, except on English-Japanese, where we validated twice per epoch. We kept the best checkpoint according to its BLEU on the development set. Inference We used beam search with a beam size of 12 for translating both the development and test sets. Since NMT often favors short translations (Cho et al., 2014), we followed Wu et al. (2016) in using a modified score s ( e | f ) in place of log-probability: s ( e | f ) = log p ( e | f ) lp ( e ) lp ( e ) = (5 + | e | )  (5 + 1)  We set  = 0 . 8 for all of our experiments. Finally, we applied a postprocessing step to replace each UNK in the target translation with the 337 source word with the highest attention score (Lu-ong et al., 2015b). Evaluation For translation into English, we report case-sensitive NIST BLEU against deto-kenized references. For English-Japanese and English-Vietnamese, we report tokenized, case-sensitive BLEU following Arthur et al. (2016) and Luong and Manning (2015). We measure statistical significance using bootstrap resampling (Koehn, 2004). 6 Results and Analysis 6.1 Overall Our results are shown in Table 3. First, we observe, as has often been noted in the literature, that NMT tends to perform poorer than PBMT on low resource settings (note that the rows of this table are sorted by training data size). Our fixnorm system alone shows large improvements (shown in parentheses) relative to tied . Integrating the lexical module ( fixnorm + lex ) adds in further gains. Our fixnorm + lex models surpass Moses on all tasks except Urduand Hausa-English, where it is 1.6 and 0.7 BLEU short respectively. The method of Arthur et al. (2016) does improve over the baseline NMT on most language pairs, but not by as much and as consistently as our models, and often not as well as Moses. Unfortunately, we could not replicate their approach for English-Japanese (KFTT) because the lexical table was too large to fit into the computational graph. For English-Japanese (BTEC), we note that, due to the small size of the test set, all systems except for Moses are in fact not significantly different from tied ( p > 0 . 01). On all other tasks, however, our systems significantly improve over tied ( p < 0 . 01). 6.2 Impact on translation In Table 4, we show examples of typical translation mistakes made by the baseline NMT systems. In the Uzbek example (top), untied and tied have confused 34 with UNK and 700 , while in the Turkish one (middle), they incorrectly output other proper names, Afghan and Myanmar , for the proper name Kenya . Our systems, on the other hand, translate these words correctly. The bottom example is the one introduced in Section 1. We can see that our fixnorm approach does not completely solve the mistranslation issue, since it translates Entoni Fauchi to UNK UNK (which is arguably better than James Chan ). On the other hand, fixnorm + lex gets this right. To better understand how the lexical module helps in this case, we look at the top five translations for the word Fauci in fixnorm + lex : e cos  W e ,  h cos  W le , h l b e + b le logit Fauci 0.522 0.762  8 . 71 7.0 UNK 0.566  0 . 009  1 . 25 5.6 Anthony 0.263 0.644  8 . 70 2.4 Ahmedova 0.555 0.173  8 . 66 0.3 Chan 0.546 0.150  8 . 73  0 . 2 As we can see, while cos  W e ,  h might still be confused between similar words, cos  W le , h l significantly favors Fauci . 6.3 Alignment and unknown words Both our baseline NMT and fixnorm models suffer from the problem of shifted alignments noted by Koehn and Knowles (2017). As seen in Figure 2a and 2b, the alignments for those two systems seem to shift by one word to the left (on the source side). For example, ni should be aligned to said instead of Telekom , and so on. Although this is not a problem per se , since the decoder can decide to attend to any position in the encoder states as long as the state at that position holds the information the decoder needs, this becomes a real issue when we need to make use of the alignment information, as in unknown word replacement (Luong et al., 2015b). As we can see in Figure 2, because of the alignment shift, both tied and fixnorm incorrectly replace the two unknown words (in bold) with But Deutsche instead of Deutsche Telekom . In contrast, under fixnorm + lex and the model of Arthur et al. (2016), the alignment is corrected, causing the UNK s to be replaced with the correct source words. 6.4 Impact of r The single most important hyper-parameter in our models is r . Informally speaking, r controls how much surface area we have on the hypersphere to allocate to word embeddings. To better understand its impact, we look at the training perplexity and dev BLEUs during training with di  erent values of r . Table 6 shows the train perplexity and best tokenized dev BLEU on Turkish-English for fixnorm and fixnorm + lex with di  erent values of r . As we can see, a smaller r results in 338 untied tied fixnorm fixnorm + lex Moses Arthur ta-en 10.3 11.1 14 ( + 2.9) 15.3 ( + 4.2) 10.5 (  0 . 6) 14.1 ( + 3.0) ur-en 7.9 10.7 12 ( + 1.3) 13 ( + 2.3) 14.6 ( + 3.9) 12.5 ( + 1.8) ha-en 16.0 16.6 20 ( + 3.4) 21.5 ( + 4.9) 22.2 ( + 5.6) 18.7 ( + 2.1) tu-en 12.2 12.6 16.4 ( + 3.8) 19.1 ( + 6.5) 18.1 ( + 5.5) 16.3 ( + 3.7) uz-en 14.9 15.7 18.2 ( + 2.5) 19.3 ( + 3.6) 17.2 ( + 1.5) 17.1 ( + 1.4) hu-en 21.6 23.0 24.0 ( + 1.0) 25.3 ( + 2.3) 21.3 (  1 . 7) 22.7 (-0.3)  en-vi 25.1 25.3 26.8 ( + 1.5) 27 ( + 1.7) 26.7 ( + 1.4) 26.2 ( + 0.9) en-ja (BTEC) 51.2 53.7 52.9 (-0.8)  51.3 (  2 . 6)  46.8 (  6 . 9) 52.4 (  1 . 3)  en-ja (KFTT) 24.1 24.5 26.1 ( + 1.6) 26.2 ( + 1.7) 21.7 (  2 . 8)  Table 3: Test BLEU of all models. Di  erences shown in parentheses are relative to tied , with a dagger (  ) indicating an insignificant di  erence in BLEU ( p > 0 . 01). While the method of Arthur et al. (2016) does not always help, fixnorm and fixnorm + lex consistently achieve significant improvements over tied ( p < 0 . 01) except for English-Japanese (BTEC). Our models also outperform the method of Arthur et al. on all tasks and outperform Moses on all tasks but Urdu-English and Hausa-English. input Dushanba kuni Hindistonda kamida 34 kishi halok bo'lgani xabar qilindi . reference At least 34 more deaths were reported Monday in India . untied At least UNK people have died in India on Monday . tied It was reported that at least 700 people died in Monday . fixnorm At least 34 people died in India on Monday . fixnorm + lex At least 34 people have died in India on Monday . input Yarn Kenya'da bir yardm konferans dzenlenecek . reference Tomorrow a conference for aid will be conducted in Kenya . untied Tomorrow there will be an Afghan relief conference . tied Tomorrow there will be a relief conference in Myanmar . fixnorm Tomorrow it will be a aid conference in Kenya . fixnorm + lex Tomorrow there will be a relief conference in Kenya . input Ammo muammolar hali ko'p , deydi amerikalik olim Entoni Fauchi . worse training perplexity, indicating underfitting, whereas if r is too large, the model achieves better training perplexity but decrased dev BLEU, indicating overfitting. One byproduct of lex is the lexicon, which we can extract and examine simply by feeding each source word embedding to the FFNN module and calculating p  ( y ) = softmax( W  h  + b  ). In Table 5, we show the top translations for some entries in the lexicons extracted from fixnorm + lex for Hungarian, Turkish, and Hausa-English. As expected, the lexical distribution is sparse, with a few top translations accounting for the most probability mass. Byte-Pair-Encoding (BPE) (Sennrich et al., 2016) is commonly used in NMT to break words into word-pieces, improving the translation of rare words. For this reason, we reran our experiments using BPE on the LORELEI and English-Vietnamese datasets. Additionally, to see if our proposed methods work in high-resource scenarios, we run on the WMT 2014 English-German (en-de) dataset, 4 using newstest2013 as the development set and reporting tokenized, case-sensitive BLEU on newstest2014 and newstest2015 . We validate across di  erent numbers of BPE operations; specifically, we try {1k, 2k, 3k} merge operations for ta-en and ur-en due to their small sizes, {10k, 12k, 15k} for the other LORELEI datasets and en-vi, and 32k for en-de. Using BPE results in much smaller vocabulary sizes, so we do not apply a vocabulary cut-o  . Instead, we train on 4 https://nlp.stanford.edu/projects/nmt/ an additional copy of the training data in which all types that appear once are replaced with UNK , and halve the number of epochs accordingly. Our models, training, and evaluation processes are largely the same, except that for en-de, we use a 4-layer decoder and 4-layer bidirectional encoder (2 layers for each direction). Table 7 shows that our proposed methods also significantly improve the translation when used with BPE, for both high and low resource language pairs. With BPE, we are only behind Moses on Urdu-English. The closest work to our lex model is that of Arthur et al. (2016), which we have discussed already in Section 4. Recent work by Liu et al. (2016) has very similar motivation to that of our fixnorm model. They reformulate the output layer in terms of directions and magnitudes, as we do here. Whereas we have focused on the magnitudes, they focus on the directions, modifying the loss function to try to learn a classifier that separates the classes' directions with something like a margin. Wang et al. (2017a) also make the same observation that we do for the fixnorm model, but for the task of face verification. Handling rare words is an important problem for NMT that has been approached in various ways. Some have focused on reducing the number of UNK s by enabling NMT to learn from a larger vocabulary (Jean et al., 2015; Mi et al., 2016); others have focused on replacing UNK s by copying source words (Gulcehre et al., 2016; Gu et al., 2016; Luong et al., 2015b). However, these methods only help with unknown words, not rare words. An approach that addresses both unknown and rare words is to use subword-level information (Sennrich et al., 2016; Chung et al., 2016; Luong and Manning, 2016). Our approach is different in that we try to identify and address the root of the rare word problem. We expect that our models would benefit from more advanced UNK replacement or subword-level techniques as well. Recently, Liu and Kirchho  (2018) have shown that their baseline NMT system with BPE already outperforms Moses for low-resource translation. However, in their work, they use the Transformer network (Vaswani et al., 2017), which is quite different from our baseline model. It would be interesting to see if our methods benefit the Trans-341 tied fixnorm fixnorm + lex ta-en 13 15 ( + 2.0) 15.9 ( + 2.9) ur-en 10.5 12.3 ( + 1.8) 13.7 ( + 3.2) ha-en 18 21.7 ( + 3.7) 22.3 ( + 4.3) tu-en 19.3 21 ( + 1.7) 22.2 ( + 2.9) uz-en 18.9 19.8 ( + 0.9) 21 ( + 2.1) hu-en 25.8 27.2 ( + 1.4) 27.9 ( + 2.1) en-vi 26.3 27.3 ( + 1.0) 27.5 ( + 1.2) en-de (newstest2014) 19.7 22.2 ( + 2.5) 20.4 ( + 0.7) en-de (newstest2015) 22.5 25 ( + 2.5) 23.2 ( + 0.7) Table 7: Test BLEU for all BPE-based systems. In this paper, we have presented two simple yet e  ective changes to the output layer of a NMT model. Both of these changes improve translation quality substantially on low-resource language pairs. In many of the language pairs we tested, the baseline NMT system performs poorly relative to phrase-based translation, but our system surpasses it (when both are trained on the same data). We conclude that NMT, equipped with the methods demonstrated here, is a more viable choice for low-resource translation than before, and are optimistic that NMT's repertoire will continue to grow. This research was supported in part by University of Southern California subcontract 67108176 under DARPA contract HR0011-15-C-0115. Nguyen was supported in part by a fellowship from the Vietnam Education Foundation. We would like to express our great appreciation to Sharon Hu for letting us use her group's GPU cluster (supported by NSF award 1629914), and to NVIDIA corporation for the donation of a Titan X GPU. We also thank Tomer Levinboim for insightful discussions.
objective objective method result other abstain abstain abstain objective objective other abstain abstain abstain abstain abstain objective objective objective abstain result method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain method other abstain other other other other method abstain other abstain other other other other method abstain other method other method abstain result objective other other other other