While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric, MACROF 1 , and study its applicability to MT evaluation. We find that MACROF 1 is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that MACROF 1 can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the meth-ods' outputs. 1 1 Introduction Model-based metrics for evaluating machine translation such as BLEURT (Sellam et al., 2020), ESIM (Mathur et al., 2019), and YiSi (Lo, 2019) have recently attracted attention due to their superior correlation with human judgments (Ma et al., 2019). However, BLEU (Papineni et al., 2002) remains the most widely used corpus-level MT metric. It correlates reasonably well with human judgments, and moreover is easy to understand and cheap to calculate, requiring only reference translations in the target language. By contrast, model-based metrics require tuning on thousands of examples of human evaluation for every new target language or domain 1 Tools and analysis are available at https://github. com/thammegowda/007-mt-eval-macro . MT evaluation metrics are at https://github.com/isi-nlp/sacrebleu/tree/ macroavg-naacl21 . (Sellam et al., 2020). Model-based metric scores are also opaque and can hide undesirable biases, as can be seen in Table 1. Reference: You must be a doctor. The source of model-based metrics' (e.g. BLEURT) correlative superiority over model-free metrics (e.g. BLEU) appears to be the former's ability to focus evaluation on adequacy , while the latter are overly focused on fluency . BLEU and most other generation metrics consider each output token equally. Since natural language is dominated by a few high-count types, an MT model that concentrates on getting its if s, and s and but s right will benefit from BLEU in the long run more than one that gets its xylophone s, peripatetic s, and defenestrate s right. Can we derive a metric with the discriminating power of BLEURT that does not share its bias or expense and is as interpretable as BLEU? As it turns out, the metric may already exist and be in common use. Information extraction and other areas concerned with classification have long used both micro averaging , which treats each token equally, and macro averaging , which instead treats each type equally, when evaluating. The latter in particular is useful when seeking to avoid results dominated by overly frequent types. In this work we take a classification-based approach to evaluating machine translation in order to obtain an easy-to-calculate metric that focuses on adequacy as much as BLEURT but does not have the expensive overhead, opacity, or bias of model-based methods. Our contributions are as follows: We consider MT as a classification task, and thus admit MACROF 1 as a legitimate approach to evaluation (Section 2). We show that MACROF 1 is competitive with other popular methods at tracking human judgments in translation (Section 3.2). We offer an additional justification of MACROF 1 as a performance indicator on adequacy-focused downstream tasks such as cross-lingual information retrieval (Section 3.3). Finally, we demonstrate that MACROF 1 is just as good as the expensive BLEURT at discriminating between structurally different MT approaches in a way BLEU cannot, especially regarding the adequacy of generated text, and provide a novel approach to qualitative analysis of the effect of metrics choice on quantitative evaluation (Section 4). Neural machine translation (NMT) models are often viewed as pairs of encoder-decoder networks. Viewing NMT as such is useful in practice for implementation; however, such a view is inadequate for theoretical analysis. Gowda and May (2020) provide a high-level view of NMT as two fundamental ML components: an autoregressor and a classifier. Specifically, NMT is viewed as a multi-class classifier that operates on representations from an autoregressor. We may thus consider classifier-based evaluation metrics. Consider a test corpus, T = {( x ( i ) , h ( i ) , y ( i ) ) i = 1 , 2 , 3 ... m } where x ( i ) , h ( i ) , and y ( i ) are source, system hypothesis, and reference translation, respectively. Let x = { x ( i )  i } and similar for h and y . Let V h , V y , V h  y , and V be the vocabulary of h , the vocabulary of y , V h  V y , and V h  V y , respectively. For each class c  V , PREDS ( c ) = m  i = 1 C ( c , h ( i ) ) REFS ( c ) = m  i = 1 C ( c , y ( i ) ) MATCH ( c ) = m  i = 1 min { C ( c , h ( i ) ) , C ( c , y ( i ) )} where C ( c , a ) counts the number of tokens of type c in sequence a (Papineni et al., 2002). For each class c  V h  y , precision ( P c ), recall ( R c ), and F  measure ( F  ; c ) are computed as follows: 2 P c = MATCH ( c ) PREDS ( c ) ; R c = MATCH ( c ) REFS ( c ) F  ; c = ( 1 +  2 ) P c  R c  2  P c + R c The macro-average consolidates individual performance by averaging by type, while the micro-average averages by token: MACROF  =  c  VF  ; c  V  MICROF  =  c  V f ( c ) F  ; c  c   V f ( c  ) where f ( c ) = REFS ( c )+ k for smoothing factor k . 3 We scale MACROF  and MICROF  values to percentile, similar to BLEU , for the sake of easier readability. In the following sections, we verify and justify the utility of MACROF 1 while also offering a comparison with popular alternatives such as MICROF 1 , BLEU , CHRF 1 , and BLEURT. 4 We use Kendall's rank correlation coefficient,  , to compute the association between metrics and human judgments. Correlations with p-values smaller than  = 0 . 05 are considered to be statistically significant. We use the 2017 WebNLG Challenge dataset (Gar-dent et al., 2017; Shimorina, 2018) 5 to analyze the differences between microand macroaveraging. WebNLG is a task of generating English text for sets of triples extracted from DBPedia. Human annotations are available for a sample of 223 records each from nine NLG systems. The human 2 We consider F  ; c for c / V h  y to be 0. 3 We use k = 1. When k   , MICROF   MACROF  . 4 BLEU and CHRF 1 scores reported in this work are computed with SACREBLEU ; see the Appendix for details. BLEURT scores are from the base model (Sellam et al., 2020). We consider two varieties of averaging to obtain a corpus-level metric from the segment-level BLEURT: mean and median of segment-level scores per corpus. 5 https://gitlab.com/webnlg/ webnlg-human-evaluation Name Fluency & Grammar Semantics BLEU  .444  .500 CHRF 1  .278 .778 MACROF 1  .222 .722 MICROF 1  .333 .611 BLEURTmean  .444 .833 BLEURTmedian .611 .667 Table 2: WebNLG data-to-text task: Kendall's  between system-level MT metric scores and human judgments. judgments provided have three linguistic aspects fluency, grammar, and semantics 6 which enable us to perform a fine grained analysis of our metrics. We compute Kendall's  between metrics and human judgments, which are reported in Table 2. As seen in Table 2, the metrics exhibit much variance in agreements with human judgments. For instance, BLEURT median is the best indicator of fluency and grammar, however BLEURT mean is best on semantics. BLEURT, being a model-based measure that is directly trained on human judgments, scores relatively higher than others. Considering the model-free metrics, CHRF 1 does well on semantics but poorly on fluency and grammar compared to BLEU . Not surprisingly, both MICROF 1 and MACROF 1 , which rely solely on unigrams, are poor indicators of fluency and grammar compared to BLEU , however MACROF 1 is clearly a better indicator of semantics than BLEU . The discrepancy between MICROF 1 and MACROF 1 regarding their agreement with fluency, grammar, and semantics is expected: micro-averaging pays more attention to function words (as they are frequent types) that contribute to fluency and grammar whereas macro-averaging pays relatively more attention to the content words that contribute to semantic adequacy. The take away from this analysis is as follows: MACROF 1 is a strong indicator of semantic adequacy, however, it is a poor indicator of fluency. We recommend using either MACROF 1 or CHRF 1 when semantic adequacy and not fluency is a desired goal. In this section, we verify how well the metrics agree with human judgments using Workshop on Machine Translation (WMT) metrics task datasets for 20172019 (Bojar et al., 2017; Ma et al., 2018, 2019). 7 We first compute scores from each MT metric, and then calculate the correlation  with human judgments. As there are many language pairs and translation directions in each year, we report only the mean and median of  , and number of wins per metric for each year in Table 3. We have excluded BLEURT from comparison in this section since the BLEURT models are fine-tuned on the same datasets on which we are evaluating the other methods. 8 CHRF 1 has the strongest mean and median agreement with human judgments across the years. In 2018 and 2019, both MACROF 1 and MICROF 1 mean and median agreements outperform BLEU whereas in 2017 BLEU was better than MACROF 1 and MICROF 1 . As seen in Section 3.1, MACROF 1 weighs towards semantics whereas MICROF 1 and BLEU weigh towards fluency and grammar. This indicates that recent MT systems are mostly fluent, and adequacy is the key discriminating factor amongst them. BLEU served well in the early era of statistical MT when fluency was a harder objective. Recent advancements in neural MT models such as Transformers (Vaswani et al., 2017) produce fluent outputs, and have brought us to an era where semantic adequacy is the focus. In this section, we determine correlation between MT metrics and downstream cross-lingual information retrieval (CLIR) tasks. CLIR is a kind of information retrieval (IR) task in which documents in one language are retrieved given queries in another (Grefenstette, 2012). A practical solution to CLIR is to translate source documents into the query language using an MT model, then use a monolingual IR system to match queries with translated documents. Correlation between MT and IR metrics is accomplished in the following steps: 1. Build a set of MT models and measure their performance using MT metrics. 2. Using each MT model in the set, translate all source documents to the target language, build an IR model, and measure IR performance on translated documents. 3. For each MT metric, find the correlation between the set of MT scores and their corresponding set of IR scores. The MT metric that has a 7 http://www.statmt.org/wmt19/metrics-task.html 8 https://github.com/google-research/bleurt Year Pairs  BLEUBLEUMACROF 1 MICROF 1 CHRF 1 2019 18 Mean .751 .771 .821 .818 .841 Median .782 .752 .844 .844 .875 Wins 3 3 6 3 5 2018 14 Mean .858 .857 .875 .873 .902 Median .868 .868 .901 .879 .919 Wins 1 2 3 2 6 2017 13 Mean .752 .713 .714 .742 .804 Median .758 .733 .735 .728 .791 Wins 5 4 2 2 6 Table 3: WMT 201719 Metrics task: Mean and median Kendall's  between MT metrics and human judgments. stronger correlation with the IR metric(s) is more useful than the ones with weaker correlations. 4. Repeat the above steps on many languages to verify the generalizability of findings. An essential resource of this analysis is a dataset with human annotations for computing MT and IR performances. We conduct experiments on two datasets: firstly, on data from the 2020 workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS) (Zavorin et al., 2020), and secondly, on data originally from Europarl, prepared by Lignos et al. (2019) (Europarl). CLSSTS datasets contain queries in English (EN), and documents in many source languages along with their human translations, as well as query-document relevance judgments. We use three source languages: Lithuanian (LT), Pashto (PS), and Bulgarian (BG). The performance of this CLIR task is evaluated using two IR measures: Actual Query Weighted Value (AQWV) and Mean Average Precision (MAP). AQWV 9 is derived from Actual Term Weighted Value (ATWV) metric (Weg-mann et al., 2013). We use a single CLIR system (Boschee et al., 2019) with the same IR settings for all MT models in the set, and measure Kendall's  between MT and IR measures. The results, in Table 4, show that MACROF 1 is the strongest indicator of CLIR downstream task performance in five out of six settings. AQWV and MAP have a similar trend in agreement to the MT metrics. CHRF 1 and BLEURT, which are strong contenders when generated text is directly evaluated by humans, do not indicate 9 https://www.nist.gov/system/files/documents-/2017/10/26/aqwv_derivation.pdf CLIR task performance as well as MACROF 1 , as CLIR tasks require faithful meaning equivalence across the language boundary, and human translators can mistake fluent output for proper translations (Callison-Burch et al., 2007). 3.3.2 Europarl Datasets We perform a similar analysis to Section 3.3.1 but on another cross-lingual task set up by Lignos et al. (2019) for Czech  English (CS-EN) and German  English (DE-EN), using publicly available data from the Europarl v7 corpus (Koehn, 2005). This task differs from the CLSSTS task (Section 3.3.1) in several ways. Firstly, MT metrics are computed on test sets from the news domain, whereas IR metrics are from the Europarl domain. The domains are thus intentionally mismatched between MT and IR tests. Secondly, since there are no queries specifically created for the Europarl domain, GOV2 TREC topics 701850 are used as domain-relevant English queries. And lastly, since there are no query-document relevance human judgments for the chosen query and document sets, the documents retrieved by BM25 (Jones et al., 2000) on the English set for each query are treated as relevant documents for computing the performance of the CS-EN and DE-EN CLIR setup. As a result, IR metrics that rely on boolean query-document relevance judgments as ground truth are less informative, and we use Rank-Based Overlap (RBO; p = 0 . 98) (Webber et al., 2010) as our IR metric. We perform our analysis on the same experiments as Lignos et al. (2019). 10 NMT models for CS-EN and DE-EN translation are trained using a convolutional NMT architecture (Gehring 10 https://github.com/ConstantineLignos/ mt-clir-emnlp-2019 Domain IR Score BLEUMACROF 1 MICROF 1 CHRF 1 BLEURTmean BLEURTmedian LT-EN In AQWV .429  .363 .508  .385 .451 .420 MAP .495 .429 .575 .451 .473 .486 In+Ext AQWV  .345 .527 .491 .491 .491 .477 MAP  .273  .455  .418  .418  .418  .404 PS-EN In AQWV .559 .653 .574 .581 .584 .581 MAP .493 .632 .487 .494 .558 .554 In+Ext AQWV .589 .682 .593 .583 .581 .571 MAP .519 .637 .523 .482 .536 .526 BG-EN In AQWV  .455 .550 .527  .382  .418 .418 MAP .491 .661 .564 .491 .527 .527 In+ext AQWV  .257 .500  .330  .404  .367  .367 MAP  .183  .426  .257  .330  .294  .294 Table 4: CLSSTS CLIR task: Kendall's  between IR and MT metrics under study. The rows with Domain=In are where MT and IR scores are computed on the same set of documents, whereas Domain=In+Ext are where IR scores are computed on a larger set of documents that is a superset of segments on which MT scores are computed. Bold values are the best correlations achieved in a row-wise setting; values with  are not significant at  = 0 . 05. BLEUMACROF 1 MICROF 1 CHRF 1 BT  BT CS-EN .850 .867 .850 .850 .900 .867 DE-EN .900 .900 .900 .912 .917 .900 Table 5: Europarl CLIR task: Kendall's  between MT metrics and RBO. BT and  BT are short for BLEURT mean and BLEURT median . All correlations are significant at  = 0 . 05. et al., 2017) implemented in the FAIRSeq (Ott et al., 2019) toolkit. For each of CS-EN and DE-EN, a total of 16 NMT models that are based on different quantities of training data and BPE hyperparame-ter values are used. The results in Table 5 show that BLEURT has the highest correlation in both cases. Apart from the trained BLEURT median metric, MACROF 1 scores higher than the others on CS-EN, and is competitive on CS-EN. MACROF 1 is not the metric with highest IR task correlation in this setting, unlike in Section 3.3.1, however it is competitive with BLEU and CHRF 1 , and thus a safe choice as a downstream task performance indicator. Unsupervised neural machine translation (UNMT) systems trained on massive monolingual data without parallel corpora have made significant progress recently (Artetxe et al., 2018; Lample et al., 2018a,b; Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2020). In some cases, UNMT yields a BLEU score that is comparable with strong 11 supervised neural machine transla-11 though not, generally, the strongest tion (SNMT) systems. In this section we leverage MACROF 1 to investigate differences in the translations from UNMT and SNMT systems that have similar BLEU . We compare UNMT and SNMT for English  German (EN-DE, DE-EN), English  French (EN-FR, FR-EN), and English  Romanian (EN-RO, RO-EN). All our UNMT models are based on XLM (Conneau and Lample, 2019), pretrained by Yang (2020). We choose SNMT models with similar BLEU on common test sets by either selecting from systems submitted to previous WMT News Translation shared tasks (Bojar et al., 2014, 2016) or by building such systems. 12 Specific SNMT models chosen are in the Appendix (Table 12). Table 6 shows performance for these three language pairs using a variety of metrics. Despite comparable scores in BLEU and only minor differences in MICROF 1 and CHRF 1 , SNMT models have consistently higher MACROF 1 and BLEURT than the UNMT models for all six translation directions. In the following section, we use a pairwise maximum difference discriminator approach to compare corpus-level metrics BLEU and MACROF 1 on a segment level. Qualitatively, we take a closer look at the behavior of the two metrics when comparing a translation with altered meaning to a translation with differing word choices using the metric. 12 We were unable to find EN-DE and DE-EN systems with comparable BLEU in WMT submissions so we built standard Transformer-base (Vaswani et al., 2017) models for these using appropriate quantity of training data to reach the desired BLEU performance. We report EN-RO results with diacritic removed to match the output of UNMT. We consider cases where a metric has a strong opinion of one translation system over another, and analyze whether the opinion is well justified. In order to obtain this analysis, we employ a pairwise segment-level discriminator from within a corpus-level metric, which we call favoritism . We extend the definition of T from Section 2 to T = { x , h S , h U , y } where each of h S and h U is a separate system's hypothesis set for x . 13 Let M be a corpus-level measure such that M ( h , y )  R and a higher value implies better translation quality. M ( h ( i ) , y ( i ) ) is the corpus-level score obtained by excluding h ( i ) and y ( i ) from h and y , respectively. We define the benefit of segment i ,  M ( i ; h ) :  M ( i ; h ) = M ( h , y ) M ( h ( i ) , y ( i ) ) If  M ( i ; h ) > 0, then i is beneficial to h with respect to M , as the inclusion of h ( i ) increases the corpus-13 The subscripts represent SNMT and UNMT in this case, though the definition is general. level score. We define the favoritism of M toward i as  M ( i ; h S , h U ) :  M ( i ; h S , h U ) =  M ( i ; h S )  M ( i ; h U ) (1) If  M ( i ; h S , h U ) > 0 then M favors the translation of x ( i ) by system S over that in system U . Table 7 reflects the results of a manual examination of the ten sentences in the DE-EN test set with greatest magnitude favoritism; complete results are in the Appendix, Tables 15 and 16. Meaning-altering changes such as untranslation' , (wrong) time' , and (wrong) translation' are marked in italics , while changes that do not fundamentally alter the meaning, such as synonym,' (different) inflec-tion,' and (different) word order' are marked in plain text. 14 The results indicate that MACROF 1 generally favors SNMT, and with good reasons, as the favored translation does not generally alter sentence meaning, while the disfavored translation does. On 14 Some changes, such as word order' may change meaning; these are italicized or not on a case-by-case basis. the other hand, for the ten most favored sentences according to BLEU , four do not contain meaning-altering divergences in the disfavored translation. Importantly, none of the sentences with greatest favoritism according to MACROF 1 , all of which having meaning altering changes in the disfavored alternatives, appears in the list for BLEU . This indicates relatively bad judgment on the part of BLEU . One case of good judgment from MACROF 1 and bad judgment from BLEU regarding truncation is shown in Table 8. From our qualitative examinations, MACROF 1 is better than BLEU at discriminating against untranslations and trucations in UNMT. The case is similar for FR-EN and RO-EN, except that ROEN has more untranslations for both SNMT and UNMT, possibly due to the smaller training data. Complete tables and annotated sentences are in the Appendix, in Section C. 5 Related Work 5.1 MT Metrics Many metrics have been proposed for MT evaluation, which we broadly categorize into model-free or model-based . Model-free metrics compute scores based on translations but have no significant parameters or hyperparameters that must be tuned a priori ; these include BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover et al., 2006), and CHRF 1 (Popovic, 2015). Model-based metrics have a significant number of parameters and, sometimes, external resources that must be set prior to use. These include METEOR (Banerjee and Lavie, 2005), BLEURT (Sellam et al., 2020), YiSi (Lo, 2019), ESIM (Mathur et al., 2019), and BEER (Stanojevic and Sima'an, 2014). Model-based metrics require significant effort and resources when adapting to a new language or domain, while model-free metrics require only a test set with references.
abstain abstain abstain abstain objective result result abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective result method objective abstain abstain abstain abstain method abstain abstain abstain other other abstain abstain method abstain abstain method abstain abstain method method abstain abstain abstain other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain method abstain abstain method abstain abstain other method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain