We propose a new automatic evaluation metric for machine translation. Our proposed metric is obtained by adjusting the Earth Mover's Distance (EMD) to the evaluation task. The EMD measure is used to obtain the distance between two probability distributions consisting of some signatures having a feature and a weight. We use word embeddings, sentence-level tf (cid:1) idf , and cosine similarity between two word embeddings, respectively, as the features, weight, and the distance between two features. Results show that our proposed metric can evaluate machine translation based on word meaning. Moreover, for distance, cosine similarity and word position information are used to address word-order differences. We designate this metric as W ord E mbedding-based automatic MT evaluation using W ord P osition I nformation (WEWPI). A meta-evaluation using WMT16 metrics shared task set indicates that our WE WPI achieves the highest correlation with human judgment among several representative metrics. Recent advances in neural machine translation (NMT) (Sutskever et al., 2014; Luong et al., 2015) are remarkable. Results based on human evaluation have demonstrated that NMT outperforms statistical machine translations significantly (Chiang, 2005; Tufis and Ceausu, 2009). The NMT achieved especially high performance in terms of fluency. However, it tends to generate more omission errors than statistical machine translations generate. Unfortunately, it is difficult for automatic evaluation metrics to evaluate outputs with omission errors because those errors are not included as non-match words between the translation and reference. For such cases, the word embedding-based automatic MT evaluation metric, which is based on word position information, is effective. Various automatic evaluation metrics have been proposed for machine translation, but none is suf-ficient for NMT. Actually, BLEU ( Papineni et al., 2002) is the representative metric based on n-gram matching. Unfortunately, because it is a surface-level metric, it is difficult to address word meaning during evaluation for MT outputs. The word-embedding-based distance measure for document ( Kusner et al., 2016) and the word-alignment-based automatic evaluation metric using word embedding ( Matsuo et al., 2017) are effective to address word meanings. Nevertheless, they can only ineffectively accommodate word order differences between the translation and reference. Given those circumstances, a new metric with word embedding-based automatic MT evaluation metric using word position information is proposed in which the evaluation score is obtained by adjusting the Earth Mover's Distance (EMD) (Rubner et al., 1998, 2000) to the evaluation task. The EMD measure represents the distance between two probability distributions. Moreover, the EMD distance is obtained based on a signature consisting of the feature and the weight, and the distance between two features. The feature, weight, and distance must therefore be defined to adjust EMD to the evaluation task. In our proposed metric, the word embeddings and the sentence-level tf (cid:1) idf respectively denote the feature and the weight. Consequently, our proposed metric can produce an evaluation based on the word meaning. Moreover, our proposed metric uses word position information in the distance between two word embeddings. The distance is obtained using cosine similarity and the difference of word position between the translation and reference. Results demonstrate that our proposed metric can evaluate translations also considering word order differences. We designate this new metric as W ord E mbedding-based automatic MT evaluation using W ord P osition I nformation (WE WPI). The experimentally obtained results based on the WMT16 metrics shared task (Bojar et al., 2016) demonstrated that our WE WPI achieves the highest correlation with human judgment among several metrics: BLEU, METEOR (Banerjee and Lavie, 2005), IMPACT (Echizen-ya and Araki, 2007), and RIBES (Isozaki et al., 2010). Moreover, the correlation of WE WPI is better than that of WEWPI without word position information (WE). Results therefore confirmed the effectiveness of WE WPI using word position information. Kusner et al. (2016) proposed the Word Mover's Distance (WMD) as a distance measure using word embedding and word alignment. This measure obtains the distance between two documents adjusting EMD to a document. However, it cannot accommodate differences of word order between the translation and reference. Matsuo et al. (2017) also proposed a word-alignment-based automatic evaluation metric using word embeddings for segment-level evaluation. As described in that paper, Maximum Alignment Similarity (MAS) was found to have higher correlation with human evaluation than BLEU for European-to-English, which has similar grammar structures. For Japanese-to-English, which has different grammar structures, Average Alignment Similarity (AAS) showed better correlation with human evaluation than other metrics. However, neither MAS nor AAS uses word position information. Therefore, neither can sufficiently accommodate word order differences. Actually, WEWPI uses not only the word alignment but also word position information. learns distributed word representations from a neural network model and from distributed sentence representations computed with a recursive autoencoder. Moreover, it uses a penalty based on translation and reference lengths. By contrast, the WE WPI system specifically examines the difference between the word positions of the translation and reference, not the difference of lengths between the translation and reference. Therefore, it can sufficiently accommodate word order differences. Moreover, it can evaluate the translation efficiently using word embeddings of target languages without requiring large amounts of data or learning time. Our WE WPI requires no learning of bilingual knowledge or a relation between translation and reference. It needs only a model of word embeddings in advance to apply EMD to the automatic MT evaluation task. In a non-trained evaluation metric, MEANT 2.0 ( Lo, 2017; Bojar et al., 2017) uses a distributional word vector model to evaluate lexical semantic similarity and shallow semantic parses to evaluate structural semantic similarity between the translation and reference. It is a new version of MEANT ( Lo and Wu, 2011), which is a non-ensemble and untrained metric. Moreover, MEANT 2.0 nosrl is a subversion of MEANT 2.0 to evaluate the translation for any output language by removing the dependence on semantic parsers for semantic role labeling (SRL). In that case, phrasal similarity is calculated using n-gram lexical similarities. However, MEANT 2.0 series do not specifically examine the position of each word in the translation and reference. Results show that it is difficult to deal sufficiently with language pairs for which the grammar differs. In WEWPI, the evaluation score is calculated using the relative difference between the positions of each word in the translation and reference. Therefore, WEWPI can evaluate translations dealing with word order in languages pairs for which the grammar differs. As described herein, we propose WEWPI as the automatic MT evaluation metric obtained by adjusting the E arth M over's D istance (EMD) to the automatic MT evaluation task. First, we describe EMD. Figure 1 depicts an outline of EMD. In Figure 1, two probability distributions are presented respectively as P and Q . The P and Q consist of some P i and Q j , which are the respective signatures. Each signature consists of a feature ( i.e. , p i in P i and q j in Q j ) and a weight ( i.e. , w p i in P i and w q j in Q j ). Therefore, two probability distributions P and Q are defined respectively as P = f ( p 1 ; w p 1 ) ::: ( p m ; w p m ) g and Q = f ( q 1 ; w q 1 ) ::: ( q n ; w q n ) g . Moreover, d ij represents the distance between two features p i and q j . The goal of EMD is to obtain total flow F = [ f ij ] that minimizes the overall cost from the perspective of a transportation problem. In that case, the overall cost is defined as Eq. (1). Moreover, four constraints are defined for f ij , which is the transportation amount in the transportation problem, to find minimum F as the following Eqs. (2)(5): f ij (cid:21) 0 1 < i < m; 1 < j < n (2) n  j =1 f ij < w p i 1 < i < m (3) m  i =1 f ij < w q j 1 < j < n (4) m  i =1 n  j =1 f ij = min 0 @ m  i =1 w p i ; n  j =1 w q j 1 A (5) Constraint (2) shows that each amount of weight f ij is transported only in the direction from signature P i to signature Q j to be nonnegative. In Constraint (3), the amount of weight which is supplied from P i ( i.e. ,  nj =1 f ij ) does not exceed w p i , which is the weight of P i . Moreover, in Constraint (4), the amount of weight which Q j receives ( i.e. ,  mi =1 f ij ) does not exceed w q j , which is the weight of Q j . Finally, the total amount of weight is equal to the weight of the lighter distribution in Constraint (5). In Eqs. (1)(5), m shows the number of signatures in P ; n shows the number of signatures in Q . The EMD is defined as shown below. In Eq. (6), the min ( W ORK ( P; Q; F )) is normalized by the minimum amount of work of Eq. (5). We describe the computation of EMD using two probability distributions P and Q in two dimensional surface. Tables 1 and 2 respectively present the examples of P and Q signatures. In Tables 1 and 2, all features p i and q j correspond to the coordinate ( x; y ) of two dimensional surface. Figure 2 depicts an example of an EMD calculation based on the signatures in Tables 1 and 2. In Figure 2, the green arrow indicates the amount of weight f ij . All f ij are transported only in the direction from P i to Q j according to Constraint (2). In each signature P i ,  nj =1 f ij does not exceed w p i by Constraint (3). For example, in P 3 ,  3 j =1 f 3 j is 0.6 (=0.2+0.0+0.4). It does not exceed 0.6, which is the weight of P 3 . Moreover, in each signature Q j ,  mi =1 f ij does not exceed w q j according to Constraint (4). For example, in Q 1 ,  4 i =1 f i 1 is 0.8 (=0.6+0.0+0.2+0.0). It does not exceed 0.8, which corresponds to the weight of Q 1 . The total amount of weight by  mi =1  nj =1 f ij is 2.4. It is equal to 2.4 by  mi =1 w p i or 2.4 by  nj =1 w q j . Therefore, this example of Figure 2 conforms to Constraint (5). Moreover, the distance between two features is necessary to obtain EMD. When the Euclidean distance is used as the calculation of distance in this example, 2.236 (= p 1 2 + 2 2 ) is obtained as d 11 , d 22 , d 31 , d 33 , d 42 , and d 43 , and other distances are 3.606 (= p 2 2 + 3 2 ) in Figure 2. As a result, 5.366 (=2.236 (cid:2) (0.6+0.6+0.2+0.4+0.2+0.4)) is obtained as the value of EMD by two probability distributions P and Q in Tables 1 and 2. We obtain WE WPI adjusting EMD to the automatic MT evaluation task. Details of application of EMD to WE WPI are presented in 3.2.2. 3.2 New Automatic MT Evaluation Metric: WE WPI 3.2.1 Word Alignment using Position Information For the application of EMD to automatic MT evaluation, we use word alignment results. Word alignment is done using cosine similarity based on word embeddings and the relative difference between the word positions in the translation and reference. In that case, WEWPI obtains align score using Eqs. In Eq. (7), t i and r j respectively represent the word embeddings of word T i in the translation and word R j in the reference. The cos sim ( t i ; r j ) denotes the cosine similarity between t i and r j . Moreover, pos inf ( T i ; R j ) represents the relative difference between the position of word T i in the translation and the position of word R j in the reference. It is defined as Eq. (8). In Eq. (8), pos ( T i ) and pos ( R j ) respectively denote the positions of word T i in the translation and word R j in the reference. Actually, m and n respectively denote the word numbers in the translation and reference. The pos inf ( T i ; R j ) becomes larger as the relative difference between pos ( T i ) and pos ( R j ) becomes larger . Therefore, (1 : 0 (cid:0) pos inf ( T i ; R j )) is used as the negative weight for cos sim ( t i ; r j ) . The ranges of cos sim ( t i ; r j ) and pos inf ( T i ; R j ) are both 0.0 1.0. Figure 3 depicts an example of word alignment using Eqs. (7) and (8). The WE WPI calculates align score between a word in the translation and all words in reference. Based on those results, the word with the highest align score in the reference is selected as the corresponding word to the word in the translation. In Figure 3, the align score between that in the translation and you in the reference is the highest ( i.e. , 0.478) among the align score between that in the translation and all words in the reference. However, it is lower than the align score 0.833 between you in the translation and you in the reference. Therefore, the word which corresponds to that in the translation cannot be obtained in the reference. Similarly, the word which 1878 reference Are there topics you want to get the world talking about ? corresponds to should in the translation cannot be obtained in the reference. In contrast, discuss in the translation corresponds to talking in the reference using pos inf ( T i ; R j ) of Eq. (8) although discuss in the translation corresponds to topics in the reference when (1 : 0 (cid:0) pos inf ( T i ; R j )) is not used in Eq. (7) ( i.e. , align score = cos sim ( t i ; r j ) ). The 0.477, which is the cos sim between discuss in the translation and topics in the reference, is greater than 0.460, which is the cos sim between discuss in the translation and talking in the reference. Here, pos inf ( T i ; R j ) between discuss in the translation and talking in the reference is 0.033 ( (cid:12)(cid:12)(cid:12) 810 (cid:0) 1012 (cid:12)(cid:12)(cid:12) ). That between discuss in the translation and topics in the reference is 0.550 ( (cid:12)(cid:12)(cid:12) 810 (cid:0) 312 (cid:12)(cid:12)(cid:12) ). Consequently, the align score of discuss in the translation and talking in the reference is 0.445 ( 0 : 460 (cid:2) (1 : 0 (cid:0) 0 : 033) ). That of discuss in the translation and topics in the reference is 0.215 ( 0 : 477 (cid:2) (1 : 0 (cid:0) 0 : 550) ) using Eq. (7). The WEWPI can select talking in the reference as the corresponding word for discuss in the translation using pos inf ( T i ; R j ) . The use of pos inf ( T i ; R j ) is effective for the correct word alignment. We obtain WEWPI as new automatic MT evaluation metrics by adjusting EMD to the automatic MT evaluation task. In WE WPI, the variables P and Q in Figure 1 respectively correspond to a translation T and reference R . Moreover, the features ( i.e. , p i and q j in Figure 1), the weight ( i.e. , w p i and w q j in Figure 1), and distance ( i.e. , d ij in Figure 1) are required as parameters to adjust EMD to the automatic MT evaluation task. As described herein, we use the word embeddings as features and the sentence-level tf (cid:1) idf as the weight. The weight definition is presented in Eq. (9) below. In Eq. (9), tf denotes the appearance frequency of a word in a translation or reference. In addition, df represents the number of sentences in which the word appears in all translations or references. In addition, N is the total number of translations or references. Actually, WE WPI distinguishes the function word and the content word using Eq. (9). Furthermore, w t i of the word in the translation and w r i of the word in the reference by Eq. (9) are normalized respectively using the following Eqs. (10) and (11). The dependence of w in Eq. (9) by difference of dataset can be kept to the minimum by normalizing Eqs. (10) and (11). Moreover, we define distance d ij , which is ascertained from the result of the word alignment described in 3.2.1. The d ij is obtained using the following Eq. (12): d ij = 8>>>< >>>: 1 : 0 (cid:0) cos sim ( t i ; r j ) (cid:2) e (cid:0) pos inf ( T i ;R j ) if T i corresponds to R j 1 : 0 if T i does not correspond to R j (12) In Eq. (12), 1 : 0 (cid:0) cos sim ( t i ; r j ) (cid:2) e (cid:0) pos inf ( T i ;R j ) is used as d ij when word T i in the translation corresponds to word R j in the reference by the word alignment result. The pos inf ( T i ; R j ) is obtained by Eq. (8). Here, t i and r j respectively correspond to the word embeddings of the words in the translation and reference. The e (cid:0) pos inf ( T i ;R j ) represents the penalty 1879 cs-en de-en fi-en ro-en ru-en tr-en Human RR DA RR DA RR DA RR DA RR DA RR DA Systems 6 6 10 10 9 9 7 7 10 10 8 8 mtevalBLEU .992 .989 .905 .808 .858 .864 .899 .840 .962 .837 .899 .895 METEOR .995 .991 .935 .887 .952 .963 .934 .909 .987 .930 .965 .980 IMPACT .997 .990 .925 .841 .908 .915 .903 .819 .962 .840 .952 .959 RIBES .995 .990 .948 .891 .894 .901 .954 .794 .972 .864 .850 .868 MEANT 2.0 (Lo, 2017) .989 .990 .947 .950 .953 .966 .940 .946 .990 .959 .980 .990 MEANT 2.0 nosrl .985 .988 .928 .942 .969 .979 .917 .930 .984 .958 .978 .987 WE .986 .976 .918 .903 .954 .963 .885 .884 .989 .938 .976 .991 WE WPI .991 .980 .958 .927 .955 .957 .919 .877 .991 .926 .977 .993 Table 4: Absolute Pearson correlation of to-English system-level metric with human assessment variants: RR, standard WMT relative ranking; DA, direct assessment of translation adequacy. to cos sim ( t i ; r j ) because it becomes smaller as pos inf ( T i ; R j ) becomes larger. As a result, d ij becomes large when the relative difference between the position of word T i in the translation and the position of word R j in the reference ( i.e. , pos inf ( T i ; R j ) ) is large. The d ij by Eq. (12) is 1.0 when word T i does not correspond to word R j . Finally, the range of d ij becomes 0.0 1.0. Moreover, the WE WPI generates the distance matrix using d ij in Eq. (12). Table 3 presents the distance matrix between the translation Are there topics that you think should discuss world? and the reference Are there topics you want to get the world talking about? in Figure 3. In Table 3, the bold typeface represents the distance between the two aligned words. The distance matrix using Eq. (12) is effective because it is not influenced by the words which are not aligned between the translation and reference. The WE WPI obtains the evaluation score by word embedding, sentence-level tf (cid:1) idf , and the distance matrix based on Eq. (12). The evaluation score of WEWPI is obtained as Eq. (13). W E W P I ( T; R ) = 1 : 0 (cid:0) min ( W ORK ( T; R; F ))  mi =1  nj =1 f ij (13) In that equation, the range of min ( WORK ( T;R;F ))  m i =1  n j =1 f ij becomes 0.0 1.0 using the weights normalized by Eqs. (10) and (11). Near 0.0, the distance between T and R is small. However, in the automatic MT evaluation metrics, the score is close to 1.0 when the evaluation for the translation is generally high. Therefore, we obtain W E W P I by taking the value of min ( WORK ( T;R;F ))  m i =1  n j =1 f ij from 1.0. As a result, in between the translation Are there topics that you think should discuss world? and the reference Are there topics you want to get the world talking about?, 0.608 is obtained as the score using Eq. (13). The WEWPI can evaluate the translation based on the meanings of words using word embedding. Moreover, it can deal with the word order using the relative difference between the positions of words in the translation and the reference. We conducted evaluation experiments to confirm the effectiveness of WEWPI. The new-stest2016 set, which is the main test set in WMT16 metrics shared task (Bojar et al., 2016), was used. The script is available at http://www.statmt.org/wmt16/results.html . Therefore, we can readily obtain the correlation coefficient between the metrics and human judgments in WMT16 metrics shared task. The WMT16 metrics task includes English paired with Czech, German, Finnish, Romanian, Russian, and Turkish. For all translations, references and scores by human judgment in these language pairs are obtained from the url described above. For these experiments, we used different automatic MT evaluation metrics for comparison with our WE WPI: BLEU, METEOR, IMPACT, RIBES, and WE. Here, IMPACT and RIBES, which are surface-based metrics, are effective for language pairs with greatly different word order, such as English and Japanese. In addition, WE is an automatic MT evaluation metric that does not perform word alignment. It uses only d ij = 1 : 0 (cid:0) cos sim ( t i ; r j ) as the d ij of Eq. (12) in the WE WPI. In both WE and WE WPI, the word vectors for seven languages ( i.e. , English, Czech, German, Finnish, Romanian, Russian, and Turkish) were obtained using fastText (Grave et al., 2018). Tables 4 and 5 respectively present the correlation coefficient of to-English and out-of-English at the system level. Tables 6 and 7 respectively present the correlation coefficients of to-English and out-of-English at the segment level. In Tables 47, RR represents the correlation based on the relative ranking by human judgment to 5 translations at a time. The bold typeface shows the highest correlation coefficient among all correlation coefficients of metrics. Moreover, the coefficients of MEANT 2.0 described in (Lo, 2017) are added to Tables 4 6. Here, WEWPI achieves the highest correlation with human judgment in Table 5, DA in Table 6, and Table 7. Especially, the correlation coefficients of WE WPI are high with language pairs for which the grammar differs ( i.e. , English-to-German (en-de), German-to-English (de-en), 1881 Figure 4: To-English system-level metric significance test of results for human assessment variants, where DA denotes the direct assessment of translation adequacy. English-to-Turkish (en-tr), and Turkish-to-English (tr-en)). Therefore, the WE WPI is effective with such language pairs because it uses word position information. Moreover, we investigated the significance of WEWPI results and those of other metrics except those of MEANT 2.0 and MEANT 2.0 nosrl. As described herein, Williams significance test (Williams, 1959) was used to assess differences in dependent correlations. Figures 49 present significance test results for every competing pair of metrics, including those of our WEWPI. However, the language pairs for which significant differences could not be obtained in any competing pair of metrics are excluded from Figures 49 ( i.e. , cs-en and fi-en in Figure 4, cs-en, fi-en and ro-en in Figure 5, en-cs in Figure 7). In Figures 49, green cells signify that the metric shows significant difference from other metrics with 95% or greater confidence. Results demonstrated that our WEWPI yielded significantly different results among metrics. Particularly, WE WPI was found to have significantly better results than those of WE at the segment level, as shown in Figures 8 and 9. This particular result demonstrates that the word position information in WEWPI is effective for segment-level evaluation. Moreover, WE WPI does not need much time to calculate the scores described in 3.2.2. However, it takes time to calculate tf (cid:1) idf of words and to change the surface-level words to the word vectors. It is efficient to calculate tf (cid:1) idf of all words in the translations and references, and to extract the word vectors, which correspond to the words in the translations and references, from the fastText models in advance. As described herein, we proposed WEWPI as a new automatic MT evaluation metric. It produces an evaluation based on the meanings of words using word embedding. Moreover, it can accommodate word-order differences. Evaluation experiments demonstrated that our WE WPI obtains the highest correlation with human judgments among several representative metrics in language pairs for which the grammar differs, and demonstrated that it is significantly better than other metrics at segment-level evaluation. Our future work will improve WE WPI to obtain high-quality evaluation scores in combination with other metrics. We will conduct evaluation experiments using various data. Moreover, we will use WEWPI to improve NMT quality. For instance, WEWPI can be used easily in Minimum Risk Training (MRT) (Shen et al., 2016), which minimizes the expected loss on the training data. This work was partially supported by grants from Hokkai-Gakuen University.
objective objective abstain method objective abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain objective objective abstain abstain result other other other other other other other other other other other objective other other abstain other other other other other other other other other objective other objective other other other other other other other other other other other other other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain objective abstain abstain objective result method result abstain other