We propose an automatic evaluation method of machine translation that uses source language sentences regarded as additional pseudo references. The proposed method evaluates a translation hypothesis in a regression model. The model takes the paired source, reference, and hypothesis sentence all together as an input. A pretrained large scale cross-lingual language model encodes the input to sentence-pair vectors, and the model predicts a human evaluation score with those vectors. Our experiments show that our proposed method using Cross-lingual Language Model (XLM) trained with a translation language modeling (TLM) objective achieves a higher correlation with human judgments than a baseline method that uses only hypothesis and reference sentences. Additionally, using source sentences in our proposed method is confirmed to improve the evaluation performance. Automatic machine translation evaluation (MTE) has been studied to substitute human evaluation in machine translation development because it is low-cost, handy, and stable to use. Popular automatic MTE metrics such as BLEU (Papineni et al., 2002) calculate the evaluation score based on a surface-level similarity of a paired 1-to-1 reference and translated hypothesis sentences. BLEU particularly evaluates the sentence similarity with the n-gram word matching rate between a reference and hypothesis. However, the evaluation score drops when a reference and hypothesis are dissimilar in the surface even if they share the same meaning. To counter this problem, METEOR (Banerjee and Lavie, 2005) is proposed to mitigate the word matching of synonyms with a synonym dictionary. Yet still, with mitigation of word matching, surface-level similarity cannot fully compensate for semantics, thus word representation instead of word symbols is used in Word Mover's Distance (Kusner et al., 2015) and bleu2vec (Tattar and Fishel, 2017). Besides, sentence representation is known to be an efficient feature instead of word representation because sentence vectors can represent more global meanings. RUSE (Shimanaka et al., 2018) and BERT (Devlin et al., 2019) based MTE, BERT regressor (Shimanaka et al., 2019), utilized sentence representation and performed well on WMT17 Metric Shared Task (Bojar et al., 2017). The metrics mentioned above compare a hypothesis translation to a reference. However, a reference translation represents only one possible translation and those MTE metrics are unlikely to correctly evaluate all candidates that share the same meanings of the reference or have fatally different meanings due to a few translation errors. This problem can be mitigated by the use of multiple reference translations as argued by Dreyer and Marcu (2012) and Qin and Specia (2015), but preparing such multiple references is costly. Hereby, we propose a method to incorporate source sentence into MTE as another pseudo reference, since the source and reference sentences should be semantically equivalent. The proposed method uses Cross-lingual Language Model (XLM) (Lample and Conneau, 2019) to handle source and target languages in a shared sentence embedding space. The proposed method with XLM trained with a translation language modeling (TLM) objective showed a higher correlation with human judgments than a baseline method using hypothesis and reference sentences. Recent advances in sentence-level embedding have been used in MTE. Shimanaka et al. (2018) proposed an MTE framework called RUSE (Regressor Using Sentence Embeddings), which uses sentence-Table 1: Available corpus size annotated with human judgments in WMT-2017 Metrics Shared Task (to-English) cs-en de-en fi-en lv-en ro-en ru-en tr-en zh-en { de,ru,tr,zh } -en all-en WMT-2015 500 500 500 -500 -1000 2000 WMT-2016 560 560 560 560 560 560 1680 3360 WMT-2017 560 560 560 560 560 560 560 2240 3920 ALL 1620 1620 1620 560 560 1620 1120 560 4920 9280 level embeddings obtained by a large-scale pretrained model like InferSent (Conneau et al., 2017), Quick Thought (Logeswaran and Lee, 2018), and Universal Sentence Encoder (Cer et al., 2018). Its regressor takes sentence vectors for a reference and translation hypothesis as inputs and returns a score, which is trained to correlate well with human evaluation (Graham et al., 2015). RUSE achieved the best correlation score with human judgments in the WMT-2017 Metrics Shared Task (Bojar et al., 2017). BERT regressor (Shimanaka et al., 2019) is a simple MTE metric based on BERT (Devlin et al., 2019) encoder. It is composed of BERT encoder and a multi-layer perceptron (MLP) regressor attached to the last layer of BERT. This BERT encoder is a 12 layers bi-directional language model, referring to BERT base ( uncased ) 1 , trained with masked language model (MLM) and next sentence prediction (NSP). BERT regressor surpassed RUSE on the WMT-2017 data. We propose an MTE method using source language sentences as additional pseudo references. We use cross-lingual language models called XLM (Lam-ple and Conneau, 2019) to encode both source and target language sentences into an embedding vector. XLM has three additional techniques to BERT: language independent subword based on Byte Pair Encoding (Sennrich et al., 2016), a language embedding layer, and a translation language modeling (TLM) objective that predicts masked words from surrounding words or a paired translation. The brief architecture of XLM is shown in Figure 1. (Lample and Conneau, 2019) reported that XLM trained with TLM objective obtains better performance than multilingual BERT(Devlin et al., 2019) on the XNLI cross-lingual classification task(Conneau et al., 2018). The proposed method has two variants for the use of source language sentences, as illustrated in Figure 2. The first one called hyp+src/hyp+ref uses two sentence-pair vectors for hypothesis-source and hypothesis-reference, encoded by a cross-lingual language model independently. These sentence-pair vectors are given to an MLP-based regression model to predict the human evaluation scores. This can be regarded as an ensemble model using a monolingual vector based on the reference and a cross-lingual vector based on the source sentence. The other one called hyp+src+ref takes a concatenation of hypotheses, source, and reference sentences as an input to a cross-lingual language model to obtain a sentence-pair vector. This sentence-pair vector is expected to be directly learned to represent the quality of the translation hypothesis given two correct sentences aligned aside. We conducted experiments to evaluate the performance of the proposed method in MTE by comparing with some existing methods. The experiments were conducted with a corpus of all language pairs to English translation from segment-level WMT2017 Metrics Shared Task (Bo-jar et al., 2017). We split sentences in WMT15 and WMT16 to training and development data with the ratio of 9:1 and whole sentences in WMT17 are used for evaluation of MTE methods. The corpus size for each language pair is shown in Table 1. We used two different models from all available XLM family models 2 : XLM15 pretrained by MLM and TLM, and XLM100 pretrained only by MLM. XLM15 is expected to perform better by the paired bilingual training of TLM, but the number of available languages is limited. XLM15 is compatible with only German, Russian, Turkish, and Chinese in the corpus, which confines the model to partial access to the corpus. On the other hand, XLM100 2 https://github.com/facebookresearch/ XLM + + + + + + + + + + + + + + + + + + + + n layer transformer encoder sentence-pair vector [/s] token 1,1 token 1,  token 1,n token 2,1 token 2,  token 2,m [/s] [/s] V [/s] V 1,1 V 1,  V 1,n V 2,1 V 2,  V 2,m V [/s] V [/s] positional embedding input tokens hidden vectors L 1 L 1 L 1 L 1 L 2 L 2 L 2 L 2 L 1 language embedding POS 0 POS 1 POS  POS n POS n+3 POS  POS n+m+2 POS n+m+3 POS n+1 [/s] L 2 POS n+2 V [/s] Figure 1: The architecture of XLM sentence-pair encoder MLP evaluation score sentence-pair encoder hypothesis + source concatenation v hyp+src, v hyp+src hypothesis + reference sentence-pair vector v hyp+src sentence-pair vector v hyp+src is compatible with all language pairs in the corpus, while it lacks supervised bilingual pretraining. Thus the experiments had two corpus settings; One was a small corpus including { German (de), Russian (ru), Turkish (tr), and Chinese (zh) } to English (en) language pairs, and the other was a whole corpus including { Czech (cz), German (de), Finnish (fi), Latvian (lv), Romanian (ro), Russian (ru), Turkish (tr), and Chinese (zh) } to English language pairs. The evaluation was conducted with Pearson's correlation to human judgments in the test set. We compared the proposed methods with SentBLEU (Bojar et al., 2017), BERT regressor (Shi-manaka et al., 2019) by our implementation. We also conducted experiments using multilingual BERT, BERT multi ( cased ) , to contrast language models and experiments limiting the model's input into source-hypothesis only and reference-hypothesis only to study the impact of adding source sentences. 2: Pearson's correlation scores in the small corpus ( { de,ru,tr,zh } -en) (MSE) loss in the training set, back-propagated to both MLP and XLM in order. The hyper-parameters were selected through grid search for the following parameters. Since models are affected by randomness in training, we ran ten experiments for each of the settings and report results of the average scores. Optimizer : { Adam }  Learning rate : { 3e-5, 1e-5, 9e-6, 7e-6 }  Number of epochs : { 1, ...,20 }  Dropout rate: { 0.1 }  Batch size : { 2, 4, 8, 16 } 4.2 Results The results of each small corpus and whole corpus experiments are shown in Tables 2 and 3, respectively. Note that XLM15 was not included in the Table 3: Pearson's correlation scores in the whole corpus (all-en) cs-en de-en fi-en lv-en ru-en tr-en zh-en avg SentBLEU 0.435 0.432 0.571 0.393 0.484 0.538 0.512 0.481 BERT regressor 0.776 0.753 0.863 0.818 0.788 0.803 0.767 0.795 multi-BERT hyp+src/hyp+ref 0.743 0.688 0.824 0.812 0.772 0.796 0.751 0.769 hyp+src+ref 0.714 0.670 0.802 0.774 0.754 0.758 0.722 0.742 hyp+src 0.599 0.525 0.699 0.681 0.586 0.633 0.571 0.613 hyp+ref 0.720 0.681 0.823 0.806 0.744 0.768 0.748 0.756 XLM100 hyp+src/hyp+ref 0.712 0.681 0.822 0.810 0.756 0.773 0.745 0.757 hyp+src+ref 0.698 0.666 0.818 0.795 0.742 0.765 0.727 0.745 hyp+src 0.510 0.531 0.672 0.662 0.543 0.602 0.537 0.580 hyp+ref 0.692 0.666 0.813 0.788 0.743 0.746 0.714 0.738 whole corpus experiment due to its limited language coverage. Performance of each language model As we can see from Table 2, the proposed method using XLM15 with hyp+src/hyp+ref structure surpassed BERT regressor in the small corpus. However, XLM100 did not work well in the experiments; its results were much worse than the others in the small corpus condition, and it did not compete with BERT regressor in the whole corpus condition as shown in Table 3. One possible reason is the lack of TLM objective pretraining in XLM100. Since the TLM task allows the model for learning semantically equivalent cross-lingual sentences directly, the TLM task can be concluded to be important for using source sentences in MTE. The results of multilingual BERT are worse than BERT regressor and XLM15, but close to XLM100 or slightly better in general. From this comparison of pretraining objectives and language models, we report that our proposed method is influenced by the multilingualism of a language model. hyp+src/hyp+ref VS hyp+src+ref The results from Table 2 and Table 3 shows that hyp+src/hyp+ref structure is better than hyp+src+ref in most of the conditions, although we expected hyp+src+ref to perform better because it can access 2 translation answers as references at the same time. This is probably because both of XLM and multilingual BERT was not pretrained to handle 3 sentences in a sequence. However, it is perhaps possible that hyp+src+ref surpasses hyp+src/hyp+ref when a fine-tuning corpus is large enough. Training data size We conducted another experiment to see the effect of the training corpus size using randomly halved { de, ru, tr, zh } -en small corpus. From the results in Table 4, BERT regressor stably performed well even when the number of training data is about 1000 sentences, however, XLM15, XLM100, and multilingual BERT deteriorated their performances. Since our proposed hyp+src/hyp+ref is an ensemble model and has a more complex network structure than hyp+ref, the Table 5: Pearson's correlation score for low and high human judgement score range in the small corpus ( { de,ru,tr,zh } -en) All DA  0 . use of XLM and multi-BERT with the proposed method requires a certain amount of training data. Therefore, our proposed method deteriorated in the halved small corpus setting. On the other hand, monolingual BERT and hyp+ref benefits from the large corpus because it has no language limitation other than English. Evaluation errors In order to see when models make errors to evaluate hypothesis sentences, we plot scatters of evaluation scores and human judgement scores (DA scores) in Figure 3(a), Figure 3(b), Figure 3(c), and Figure 3(d). Although in comparison, the evaluation scores of our best model XLM15 hyp+src/hyp+ref are set more linearly than the baseline BERT regressor, the scores of all models seem much dispersed in the low DA area (DA < 0 . 0 ). This indicates that all evaluation models listed here tend to miss-evaluate when a hypothesis is poor. Furthermore, we show Pearson's correlation score for each of high and low DA score range in Table 5. As we confirmed in the scatter figures, the correlation scores of low DA is low; evaluation models work poorly when hypotheses are poor. However, the reduction rate of Pearson's scores from high DA to low DA is small with XLM15 hyp+src/hyp+ref and hyp+src. Therefore adding source sentences has an impact to stabilize the evaluation performance when hypotheses are low-quality. that the proposed method with TLM-trained XLM showed a higher correlation with human judgments than the baseline method in the small corpus condition and stabilize the evaluation performance regardless of the quality of translation sentences by using additional source sentences. We also investigated why our proposed method worked poorly in the other conditions and found the importance of TLM training. In future work, we will work around the problem of evaluation errors in the low DA range. This work is supported by JST PRESTO (JP-MJPR1856).
objective abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain other other other other other other other other objective method other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain other