Neural machine translation (NMT) encodes the source sentence in a universal way to generate the target sentence word-by-word. However, NMT does not consider the importance of word in the sentence meaning, for example, some words (i.e., content words) express more important meaning than others (i.e., function words). To address this limitation, we first utilize word frequency information to distinguish between content and function words in a sentence, and then design a content word-aware NMT to improve translation performance. Empirical results on the WMT14 English-to-German, WMT14 English-to-French, and WMT17 Chinese-to-English translation tasks show that the proposed methods can significantly improve the performance of Transformer-based NMT. Neural machine translation ( NMT ) models (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) often utilize the global neural networks to encode all words for learning the sentence representation and the context vector, and computes the accuracy of each generated target word in a universal manner. Meanwhile, each generated target word makes the same contribution to the optimization of the NMT model, regardless of its importance. Actually, there lacks a mechanism to guarantee that NMT captures the information related to word importance when predicting translations. Intuitively, content words express more important meanings than function words, which indicates their comparative significance. To evaluate this, we randomly masked content or function words with UNK in a source sentence. Figure 1 shows that the BLEU scores of the test set decreased much  Corresponding author 0 1 2 3 4 5 6 7 8 9 10 10 20 30 Number BLEU Transformer (base) -Function Words -Content Words Figure 1: Number denotes the number of content or function words that were randomly masked in each sentence of the WMT14 English-to-German translation task. more substantially when parts of content words were randomly replaced with UNK on the WMT14 English-to-German task, which is in line with the findings in He et al. (2019)'s work. To address this limitation, we propose a content word-aware NMT model that exploits the results of translation using a sequence of content words learned by a simple content word recognition method. Inspired by the works of (Setiawan et al., 2007, 2009; Zhang and Zhao, 2013), we first divide words in a sentence into content words and other function words depending on term frequency-inverse document frequency ( TF-IDF ) constraints. Two methods are designed to utilize the sequence of content word on the source and target sides: 1) We encode the content words of the source sentence as a new source representation, and learn an additional content word context vector based on it to improve translation performance; 2) A specific loss for content words of the target sentence is introduced to compensate for the original training objection, to obtain a content word-aware NMT model. Empirical results on the WMT14 English-to-German, WMT14 English-to-French, and WMT17 Chinese-to-English tasks show the effectiveness of the proposed method. In Transformer-based NMT (Vaswani et al., 2017), the encoder is composed of a stack of L identical layers, each of which contains two sub-layers. The first sub-layer is a self-attention module (ATT), and the second sub-layer is a position-wise fully connected feed-forward network (FNN). A residual connection (He et al., 2016) is applied between the sub-layers, and layer normalization (LN) (Ba et al., 2016) is performed. Formally, the l -th identical layer of this stack is as follows: H l = LN ( ATT le ( Q l  1 e , K l  1 e , V l  1 e ) + H l  1 ) H l = LN ( FFN le ( H l ) + H l ) . { Q l  1 e , K l  1 e , V l  1 e } are query, key, and value vectors that are transformed from the ( l -1)-th layer H l  1 . For example, { Q 0 , K 0 , V 0 } are packed from the H 0 learned by the positional encoding mechanism (Gehring et al., 2017). Similarly, the decoder is composed of a stack of L identical layers. Compared with the stacked encoder, it contains an additional attention sublayer to compute alignment weights for the output of the encoder stack HL : S li = LN ( ATT ld ( Q l  1 i , K l  1 i , V l  1 i ) + S l  1 i ) , C li = LN ( ATT lc ( S li , K Le , V Le ) + S li ) , S li = LN ( FFN ld ( C li ) + C li ) , (2) where Q l  1 d , K l  1 d , and V l  1 d are query, key, and value vectors, respectively, that are transformed from the ( l -1)-th layer S l  1 in time-step i . { K Le , V Le } are transformed from the L -th layer of the encoder. The top layer of the decoder S Li is used to generate the next target word y i by a linear, potentially multi-layered function: P ( y i | y <i , x )  exp ( W o tanh ( W w S Li ) , (3) where W o and W w are projection matrices. To obtain the translation model, the training objection maximizes the conditional translation probabilities over the training data set { [ X , Y ] } : J (  ) = arg max  { P ( Y | X ;  ) } . content word recognition method based on the TF-IDF (Chen et al., 2019; Zhang et al., 2020). An input sentence of length J m is treated as a document D m , and the TF-IDF T I j for each word d j in D m is computed: T I j = k j,m J m  log | M | 1 + | m : d j  D m | , (5) where k j,m represents the number of occurrences of the j -th word in the input sentence d t ; | M | is the total number of sentences in the monolingual data; and | m : d j  D m | is the number of sentences including word d j in the monolingual data. We then select a fixed percentage N (30% in the experiment) of word with high TF-IDF scores in the sentence as content words. Note that we focus on statistics related to word frequency here, instead of the linguistic criteria; this method of approximation eliminates the need for additional language-specific resources. In this section, we propose two ways to make use of the information on content words, designing three content word-aware NMT models. The proposed method of content word recognition is first added as an additional module to the encoder to learn the sequence of source content words X from the input source sentence. X is mapped and fed into the shared encoder (Li et al., 2020) in Eq.(1) to learn an additional source representation of content words HL . An multi-head attention module is then introduced to the decoder to learn the context vector C li based content words at time-step i , and C li is used to enhance the output S li : S li = LN ( ATT ld ( Q l  1 i , K l  1 i , V l  1 i ) + S l  1 i ) , C li = LN ( ATT lc ( S l i , K Le , V Le ) + S l i ) , C li = LN ( ATT ly ( S li , K Le , V Le ) + S li ) , S li = LN ( FFN ld ( C li + C li ) + C li ) , (6) where K Le and V Le of the content words are transformed from the L -th layer of the encoder. Finally, the top layer of the decoder S Li , which is enhanced by the contextual vector of the content words C ld , is used as input to the Eq. (3) to compute the probabilities of the next target word y i at time-step i : P ( y i | y <i , x )  exp ( W o tanh ( W w S Li ) . Note that both the original source representation HL and proposed content word based representation HL are learned by a shared encoder using our content word recognition module. Like the source sentence, the target sentence also contains content words. We thus first identify a sequence of content words b from the target reference translation y according to the proposed content word recognition method (see Section 3). We then introduce an addition loss term as a measure of the content words, which encourages the translation model to attend to the translation of the content words. Formally, the training objective is revised as: J (  ) = arg max  { P ( y | x ;  )+   P ( b | x ;  ) } , (8) where  is a hyper-parameter empirically set to 0.4 in this paper. Note that the introduced content word-aware loss works without any new parameters and influences only the computation of loss during the training of the standard NMT model. Based on the above two strategies, we design three NMT models: 1) SCWAContext : The source content words are used to learn an additional context vector to improve the prediction of target word (see Figure 2(a)); 2) TCWALoss : The target content words are used to compute an additional loss to guide the training of the translation model (see Figure 2(b)); 3) BCWAContLoss : It combines SCWAContext and TCWALoss to capture the content words of both the source and the target sentence to further improve translation performance. The proposed methods were evaluated on the WMT14 English-to-German (EN-DE), WMT14 English-to-French (EN-FR), and WMT17 Chinese-to-English (ZH-EN) tasks. The EN-DE corpus consists of 4M sentence pairs, the ZH-EN corpus of 22M sentence pairs, and the EN-FR corpus of 36M sentence pairs. We used the case-sensitive 4-gram BLEU score as evaluation metric. The results of the newstest2014 test sets are reported for the EN-DE and EN-FR tasks, and the newstest2017 test set is reported for the ZH-EN task. The byte pair encoding algorithm (Sennrich et al., 2016) was applied to encode all sentences to limit the size of the vocabulary to 40K. The other configurations were identical to those in (Vaswani et al., 2017). The poposed models were implemented by using Systems EN-DE ZH-EN EN-FR BLEU #Speed #Param BLEU #Param BLEU #Param Existing NMT systems Trans.base (Vaswani et al., 2017) 27.3 N/A 65.0M N/A N/A 38.1 N/A +Context-Aware SANs (Yang et al., 2019a) 28.26 N/A 106.9M 24.67 126.8M N/A N/A +Convolutional SANs (Yang et al., 2019b) 28.18 N/A 88.0M 24.80 N/A N/A N/A +BIARN (Hao et al., 2019) 28.21 N/A 97.4M 24.70 107.3M N/A N/A Trans.big (Vaswani et al., 2017) 28.4 N/A 213.0M N/A N/A 41.0 N/A +Context-Aware SANs (Yang et al., 2019a) 28.89 N/A 339.6M 24.56 379.4M N/A N/A +Convolutional SANs (Yang et al., 2019b) 28.74 N/A 339.6M 25.01 N/A N/A N/A +BIARN (Hao et al., 2019) 28.98 N/A 333.5M 25.10 373.3M N/A N/A Our NMT systems Trans.base 27.48 13.2K 66.5M 24.28 74.7M 38.32 66.9M +SCWAContext 28.28+ 12.1K 72.8M 24.79+ 81.0M 39.41+ 73.2M +TCWALoss 27.94+ 14.3K 66.5M 24.65 74.7M 38.89+ 66.9M +BCWAContLoss 28.51+ 13.1K 72.8M 24.94+ 81.0M 39.56+ 73.2M Trans.big 28.45 11.2K 221.1M 24.55 237.5M 41.21 222.9M +BCWAContLoss 29.14+ 10.1K 246.3M 25.12+ 262.7M 42.57+ 247.1M Table 1: Results of the EN-DE, EN-FR, and ZH-EN tasks. Table 1 shows results of the proposed method over our implemented Trans.base/big models which have similar BLEU scores with the original Transformer for the EN-DE and EN-FR tasks. We then make the following observations: 1) All proposed three word-aware NMT models outperformed the baseline Transformer model. This indicates that using information on the importance of words to enhance the translation of content words is helpful for the NMT model. 2) +SCWAContext performed better than +TCWALoss. The NMT model was more sensitive to information on source content words than target content words. +BCWAContLoss outperformed +SCWAContext and +TCWALoss, especially is superior to the existing +Context-Aware, +CSANs, and +BIARN. This suggests that the sequences of content words of both source and the target can be used together to further improve translation performance. 3) The parameters of the proposed models only slightly increased. In addition, Trans.base+BCWAContLoss delivered an comparable performance to Trans.big, which contained many more parameters than Trans.base+BCWAContLoss. This indicates that the improvement in performance did not occur owing to a greater number of parameters. The training speeds of our models were slightly lower than those of Trans.base. Figure 3 shows the results of the Trans.base+SCWACont based different percentage N of content words in a sentence on the EN-DE and ZH-EN test sets. On both test sets, the highest BLEU scores were obtained with N = 30%. With increasing values of N , the trend of their BLEU scores were similar on both test sets. The percent of N in the content word recognition method Figure 3: Results of Trans.base+SCWAContext model on the EN-DE and ZH-EN test set. The dashed line denotes the Trans.base model. We apply the proposed content word recognition method to the generated translation and the reference translation of test set, and thus extract two short sequences of including 30% of content words. We compute the accuracy of unigram content word between the extracted two short sequences, as shown in Table 2. The proposed methods outperformed the Trans.base in translating the content words, which is in line with the BLEU. This means that the proposed NMT model improved the generation of target content words. Figure 4 shows the results of +TCWALoss model on the EN-DE and ZH-EN test sets with different hyper-parameter  . When  increased from 0 to 0.4, the BLEU scores of +TCWALoss model improved by +0.8 points over Trans.base model. This means that the proposed content word-aware loss is useful for training NMT model. Subsequently, larger values of  reduced the BLEU scores, suggesting that excessive biased content word translation may be weak at translating function words. Therefore, we set the hyper-parameter  to 0.4 to control the loss of target content words in our experiments (Table 1). Instead of directly identify content words, we identify the function words as the T most frequent words in the corpus. Furthermore, after we remove the function words in a sentence x = { x 1 ,    , x J } , all the remaining words will be treated as a sequence (maintain the original order) of content words X according to the (Setiawan et al., 2007, 2009; Zhang and Zhao, 2013)'s work. Figure 5 shows the results of Trans.base+SCWAContLoss on the EN-DE and ZH-EN test sets with different number of the top T function words. Trans.base+SCWAContLoss obtained the highest BLEU scores on the both test sets over the Trans.base on modeling T = 256. The number of function words TBLEUEN-DE Figure 5: BLEU scores of Trans.base+SCWAContLoss on the EN-DE and ZH-EN test sets with different number of function words T . This paper explored the importance of word for NMT. We divided words of one sentence into content and function words through word frequency-related information. Our proposed NMT models, that are easy to implement and not much time and space cost, are introduced to the training and inference, and can improve the representation and translation of content words. In future work, we will investigate the impact of fine-grained word categories (such as nouns, verbs, and adjectives) on the translation performance and design specific methods according to these categories. We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. Masao Utiyama is partly supported by JSPS KAKENHI Grant Number 19H05660. Rui Wang was partially supported by JSPS grant-in-aid for early-career scientists (19K20354): Unsupervised Neural Machine Translation in Universal Scenarios and NICT tenure-track researcher startup fund Toward Intelligent Machine Translation.
abstain abstain objective abstain abstain abstain abstain abstain method abstain abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method objective objective other other other