Lexical ambiguity poses one of the greatest challenges in the field of Machine Translation. Over the last few decades, multiple efforts have been undertaken to investigate incorrect translations caused by the polysemous nature of words. Within this body of research, some studies have posited that models pick up semantic biases existing in the training data, thus producing translation errors. In this paper, we present DIBIMT, the first entirely manually-curated evaluation benchmark which enables an extensive study of semantic biases in Machine Translation of nominal and verbal words in five different language combinations, namely, English and one or other of the following languages: Chinese, German, Italian, Russian and Spanish. Furthermore, we test state-of-the-art Machine Translation systems, both commercial and non-commercial ones, against our new test bed and provide a thorough statistical and linguistic analysis of the results. We release DIBIMT at https:// nlp.uniroma1.it/dibimt as a closed benchmark with a public leaderboard. The polysemous nature of words poses a longstanding challenge in a wide range of Natural Language Processing (NLP) tasks such as Word Sense Disambiguation (Navigli, 2009; Bevilacqua et al., 2021) (WSD), Information Retrieval (Krovetz and Croft, 1992) (IR) and Machine Translation (Emelin et al., 2020) (MT). In MT, some research works have addressed the ability of systems to disambiguate polysemous words. For instance, given the sentence He poured a shot of whiskey , the polysemous target word shot unequivocally means a small quantity and therefore a possible translation into Italian could be: Vers un goccio di whiskey . However, some MT systems propose the following translation: Vers uno sparo  Equal contribution. di whiskey in which the noun sparo means gunshot . This is one of many examples that seem to encourage a deeper performance analysis in scenarios in which MT systems are required to deal with polysemous words and, specifically, with infrequent meanings of polysemous words. Although state-of-the-art MT systems, both commercial and non-commercial ones, achieve impressive BLEU scores on standard benchmarks, in our work we demonstrate that they still present significant limitations when dealing with infrequent word senses, which standard metrics fail to recognize. In the last few decades, attempts have been made to investigate the aforementioned phenomena. In fact, recent studies have observed a direct correlation between semantic biases in the training data and semantic errors in translation. However, their findings are limited by the following shortcomings: i) they are not based on entirely manually-curated benchmarks; ii) they rely heavily on automatically-generated resources to determine the correctness of a translation; and iii) they do not cover multiple language combinations. In this work, we address the aforementioned drawbacks and present DIBIMT, to the best of our knowledge the first fully manually-curated evaluation benchmark aimed at investigating the impact of semantic biases in MT in five language combinations, covering both nouns and verbs. This benchmark allows the community not only to better explore the described phenomena, but also to devise innovative MT systems which better deal with lexical ambiguity. Specifically, the contributions of the present work are threefold:  We present DIBIMT, a novel gold-quality test bed for semantic biases in MT that goes beyond a simple accuracy score, covering five language combinations, namely English and one or other of the following languages: Chinese, German, Italian, Russian and Spanish; He poured a shot of whiskey. goccio bicchierino iniezione sparo trago chupito pistolero tiro SPANISH  a   RUSSIAN    CHINESE Schlckchen Schuss Injektion Schlag GERMAN ITALIAN Figure 1: Example of an annotated dataset item. We define four novel metrics that better clarify the semantic biases within MT models;  We provide a thorough statistical and linguistic analysis in which we compare 7 state-of-the-art MT systems, including both commercial and non-commercial ones, against our new benchmark. Furthermore, we extensively discuss the results. To enable further research, we release DIBIMT as a closed benchmark with a public leaderboard at https://nlp.uniroma1.it/dibimt . Over the course of the last few decades, several approaches to the evaluation of the lexical choice in MT have been proposed. To this end, cross-lingual benchmarks were created in which systems were required to provide the translation or a substitute for a given target word in context in a target language (Vickrey et al., 2005; Mihalcea et al., 2010; Lefever and Hoste, 2013). More recently, Gonzales et al. (2017) put forward ContraWSD, a dataset which includes 7,200 instances of lexical ambiguity for German  English, and 6,700 for German  French. This dataset pairs every reference translation with a set of contrastive examples which contain incorrect translations of a polysemous target word. For each instance, the answer provided by systems is considered correct if the reference translation is scored higher. Based on a denoised version of the ContraWSD dataset and focusing on the language combination German  English, Gonzales et al. (2018) present the Word Sense Disambiguation Test Suite which, unlike ContraWSD, evaluates MT output directly rather than by scoring translations. The suite consists of a collection of 3,249 sentence pairs in which the German source sentences contain one ambiguous target word. As target words, the authors considered only words in German whose translation into English does not cover multiple senses, thus making the evaluation more straightforward. Despite their effectiveness, such benchmarks do not allow systems to be tested in multiple language combinations, and only cover a very limited number of words and senses. To address these limitations, Raganato et al. (2019) proposed MuCoW, an automatically-created test suite covering 16 language pairs, with more than 200,000 sentence pairs derived from word-aligned parallel corpora. Other research studies investigated the disambiguation capabilities of MT systems by exploring their internal representations (Marvin and Koehn, 2018; Michel et al., 2019), or improving them via context-aware word embeddings (Liu et al., 2018). More recently, Emelin et al. (2020) introduced a statistical method for the identification of disambiguation errors in neural MT (NMT) and demonstrated that models capture data biases within the training corpora, which leads these models to produce incorrect translations. Although the authors expected their approach to be transferable to other language combinations, they only focused on German  English. Based on the findings and open research questions raised in the aforementioned works, the present paper aims at investigating not only the presence, but also, most importantly, the nature and properties of semantic biases in MT in multiple language combinations, via a novel entirely manually-curated benchmark called DIBIMT and a thorough performance analysis. The DIBIMT benchmark focuses on detecting Word Sense Disambiguation biases in NMT, i.e., biases of certain words towards some of their more frequent meanings. The creation of such a dataset requires i) a set of unambiguous and grammatically-correct sentences containing a polysemous target word; ii) a set of correct and incorrect translations of each target word into the languages to be covered. Figure 1 depicts an example of a dataset item. BabelNet Similarly to previous studies, we rely on BabelNet 1 (Navigli et al., 2021), a large multilin-1 gual encyclopedic dictionary whose nodes are concepts represented by synsets, i.e., sets of synonyms, containing lexicalizations in multiple languages and coming from various heterogeneous resources, including, inter alia, WordNet (Miller et al., 1990) and Wiktionary. 2 Let us define B as an abstraction used to query the subset of synsets in BabelNet that contain at least one sense 3 from WordNet and one or more senses in languages other than English, 4 while only considering senses coming from high-quality sources, i.e., language-specific wordnets. Formal Notation Given an arbitrary synset  , we define  L (  ) as the set of lexicalizations of  in language L contained within B . As an example, let us consider the synset   corresponding to the drink meaning of the word shot . contains lexicalizations in different languages, including: Shot DE , shot EN , nip EN , chupito ES , trago ES , bicchierino IT and goccio IT . Hence,  EN (  ) = { shot , nip } , while  ES (  ) = { chupito , trago } . Furthermore, let  P represent a (lemma, part of speech) pair, where P is the part of speech. We denote  L (  P ) = {  1 , . . . ,  n } as the set of synsets which contain  P as a lexicalization in language L according to B . Additionally, we define  L (  P ) = |  L (  P ) | as the polysemy degree, i.e., the number of senses, of  P in language L . For example, given  P = shot NOUN ,  EN (  P ) would be the set of synsets associated with the nominal term shot (e.g., the act of firing, a photograph and a drink, among others). In this section, we detail the creation process of our dataset, i.e., the selection of our sentences as well as the construction and filtering of our items. Item Structure and Notation Before we proceed, let us formally state how each item in the dataset is structured: given a source sentence s = [ w 1 , . . . , w n ] as a sequence of words, and given a target word 5 w i in s tagged with some synset  , we consider X = ( s, w i ,  ) as an initial item of the dataset, i.e., an instance composed of 2 https://www.wiktionary.org/ 3 A sense is a lexicalization of a specific synset in some language. Henceforth, we will refer to lexicalizations and senses interchangeably. 4 Specifically, we consider synsets that have lexicalizations in English, Italian, German, Russian, Spanish and Chinese. 5 For simplicity, we use the term word here, but our work focuses on multi-word expressions as well (both in source and target sentences). an English sentence s , a target word w i and its associated synset  ; this instance can be annotated for candidate translations of w i in some language L . We also denote  XP as the (lemma, POS) pair of w i . We collect our initial items from two main sources: WordNet and Wiktionary. 6 Specifically, we use the examples from WordNet Tagged Glosses (Langone et al., 2004), where each sentence's target word was manually associated with its synset 7 , thereby readily providing the first batch of initial items . As for Wiktionary, instead, we start by obtaining every usage example s and its associated definition d (filtering out archaic usages and slang), then, we automatically extract the target words from the corresponding example. 8 Now, the only step that remains in order to construct an initial item is to associate a synset  with the word w i used in the example s . We perform this association in two phases: first, we try to map the definition d related to the example s to a BabelNet synset by relying on the automatic mappings available in BabelNet 5 between WordNet and Wiktionary, discarding examples for which this association can not be found; second, we manually validate and correct these successful associations to ensure that our initial items are of high quality. We apply a filtering step to the original sentences in order to select examples that are likely to be more challenging for the models to translate: i) we discard every initial item X for which  EN (  XP ) < 3 , i.e., we retain only sentences whose associated (lemma, POS) pair has a polysemy degree of at least 3 in BEN ; ii) we retain at most only one sentence per sense per source 9 ; iii) differently from previous works, which impose a strict requirement on synsets that are monosemous in the target language, we retain sentences satisfying the following requirement. Let us consider the nominal senses of the word bank : among them, one represents a specific aviation maneuver. In Italian, this synset 6 We use the dump of September 2021. 7 Which we convert from WordNet to BabelNet. 8 In Wiktionary, target words are marked in bold inside the example sentence. 9 The reasoning for this choice is twofold: on the one hand, oftentimes Wiktionary has multiple examples for the same synset, that differ in only one or two words, thus we skip them to avoid repetitions; on the other hand, we obtain an increase in sense coverage without worsening the annotator load. includes one lexicalization, avvitamento ; although this is not monosemous in Italian (e.g., avvitamento might also refer to a screw thread ), neither of the other possible senses of avvitamento has bank as an English lexicalization, which, for Italian, satisfies our third condition. If the same holds true for all languages, the synset passes the test and thus the sentence is retained. Once the set of initial items is ready, we can proceed with the annotation phase, which will produce our annotated items . Specifically, given a language L and an initial item X = ( s, w i ,  ) , we associate a set of good ( GL ) and bad ( BL ) translation candidates with X , which represent words that, respectively, we do, and do not, expect to see in a translation of sentence s in language L . Finally, we refer to XL as an annotated item , i.e., the tuple ( s, w i , , GL , BL ) . Before moving forward with the annotation phase, we pre-populate the sets of good ( GL ) and bad ( BL ) lexicalizations for a given initial item X in language L extracting them from B . Formally, we assign GL =  L (  ) , i.e., the set of lemmas in language L of the BabelNet synset associated with  ; furthermore, we set BL = S     L (  XP ) \{  }  L (  ) , i.e., the set of all lemmas in language L of BabelNet synsets associated with any   excluding  . With this step, we produce an automatically populated version of our annotated items . We instruct annotators to update the set of good ( GL ) and bad ( BL ) lexicalizations of w i  s such that each lexicalization contained in the respective set can be considered a good or a bad translation equivalent for the target word in the provided sentential context. 10 We also instruct annotators to discard sentences in which i) the target word w i is an idiomatic expression or a proper noun, and ii) the semantic context is not sufficient to properly disambiguate w i . Given the expertise required to carry out this task, we rely on three highly qualified translators: one for Italian, German and Russian; one for Spanish and one for Chinese. Our annotators satisfy the 10 Any lexicalization of  in L that is removed from GL is automatically placed in BL . following requirements: they are native speakers or hold C2-level certifications and work as professional translators in the given language combinations. The full instructions provided to the annotators can be found in Appendix C. 3.3.3 Resulting Dataset Our annotators analyzed around 800 sentences, discarding 200 of them, finally obtaining approximately 600 annotated items in 5 languages. Due to a coverage issue of the Russian language in BabelNet, we retain only sentences tagged with nominal or verbal synsets. Dataset statistics are reported in Table 1. As expected, we note that the lexicalizations found in B have been substantially refined by our annotators in all languages, as reported in Table 2. Indeed, across languages, on average, 54% of the good lexicalizations have been added by our annotators, while 42% of the pre-existing lexicalizations have been removed. More importantly, given a language and two sentences containing words referring to the same synset, on average only in 55% of cases do they also share those words' good lexicalizations, confirming that the assumption that all synonyms of a word are valid replacements can lead to incorrect results. These statistics lead us to a straightforward, but important, conclusion: only in a limited number of cases is a lexicalization belonging to a given synset to be considered as a suitable translation equivalent for the provided target word and its context. Examined jointly, these metrics suggest that relying on synset lexicalizations from BabelNet alone is prone to producing errors, either due to BabelNet's intrinsic noise, or due to the lack of different granularity of synsets and contextualized words. Sentences' Properties Description As we stated in Section 3.2.1, the sentences we annotate are all usage examples of specific concepts obtained from WordNet or Wiktionary. Such examples are typically short main clauses with no subordinates, featuring on average 9 words (around 50 characters per sentence). All selected sentences include a semantic context which allows the meaning of the target word to be properly identified. DIBI MT's analysis procedure is fairly simple: given an annotated item XL = ( s, w i , , GL , BL ) and a translation model M , we compute t L = ML ( s ) , i.e., the translation of s in language L according to M . Then, we use Stanza (Qi et al., 2020) to perform tokenization, part-of-speech tagging and lemmatization of t L and, finally, we check if there is any match 11 between the lemmas of the translated sentence and those contained in GL or BL . In case there is no match, we mark the translation as a MISS ; otherwise, we mark it as GOOD or BAD depending on which set matched the lemma. This produces an analyzed item , which for simplicity we denote as XM L = ( XL , t L , R ,  L ) , where R is one of GOOD , BAD or MISS and  L represents the matched lemma in case there was a match ( GOOD or BAD ),  otherwise. We now: i) use DIBIMT to carry out an evaluation of 7 different machine translation systems; ii) report the obtained results, including a thorough statistical and linguistic evaluation; iii) extensively discuss our findings, providing multiple measures 11 A more detailed description of the analysis procedure is provided in Appendix A. of semantic bias; and iv) offer some insights into the causes of such biases. In Appendix D we include a model-specific breakdown of the various scores and metrics reported throughout this section. We test a wide range of models, both commercial and non-commercial ones, and report their performances on DIBI MT's evaluation metrics: DeepL Translator 12 , a state-of-the-art commercial NMT system. Google Translate 13 , arguably the most popular commercial NMT system. OPUS (Tiedemann and Thottingal, 2020), the smallest state-of-the-art NMT model available to date, a base Transformer (each model has approximately 74M parameters) trained on a single language pair on large amounts of data. MBart50 (Tang et al., 2021), multilingual BART fine-tuned on the translation task for 50 languages (610M parameters). We refer to MBart50 as the English-to-many model, and to MBart50 MTM as the many-to-many model. M2M100 (Fan et al., 2021), a multilingual model able to translate from/to 100 languages. We test both versions of the model, the 418M parameter one (which we dub M2M100 ) and the 1.2B parameter one (dubbed M2M100 LG ). Figure 2 reports general results of the analysis per (model, language) pair. Given the high percentage of analyzed items classified as MISS , we asked our annotators to perform an inspection on a random sample of 70 items per language in order to unearth the reasons, with varying results. We identified multiple causes, namely: i) word omission in the translation (around 19% of items, mostly in Chinese and Italian); ii) issues with Stanza's tokenization (around 11%, mostly Chinese and Russian) and lemmatization (around 12%, mostly Italian and German); iii) words translated as themselves (approximately 5%, often in multilingual neural models); iv) translations which have nothing to 12 https://deepl.com/ 13 We used the =GOOGLETRANSLATE function available in Google Sheets. do with the source text 14 (around 23%); and v) missing terms from either BL (around 18%) or GL (around 11%). We intend to thoroughly investigate and tackle these issues and translation phenomena as future work. Table 3 reports accuracy for nonMISS analyzed items (i.e., # GOOD # GOOD +# BAD ). With the sole exception of DeepL, which greatly outperforms every other competitor, models achieve extremely low scores, in the range of 20%-33%. Surprisingly, Google Translate performs worst across languages. In addition to accuracy, DIBIMT analyzes the semantic biases of a translation model via four novel metrics, which we define in detail in what follows. Sense Frequency Index Influence (SFII) We study the sensitivity of models to disambiguating senses with respect to their frequency. To do this, we define   P (  ) as the index of synset  in  EN (  P ) ordered according to WordNet's sense frequency, as computed from SemCor. That is, in-14 An example is the sentence he is a crack shot , where the word shot is translated by MBart50 into Italian as schianto, which can be interpreted in this case as someone very good looking. dex k means that synset  is the k -th most frequent meaning for  P . In Figure 3(a), we plot the number and percentage of errors made on average by the models, grouping items by   XP (  X ) , where X is a nonMISS analyzed item . As expected, the less frequent a meaning for a given word is, the harder it is for the model to correctly disambiguate it. Finally, given a (model, language) pair, we define the Sense Frequency Index Influence (SFII) as the average percentage of errors, for each group, that we detected. Values are reported in Table 4. Interestingly, DeepL proves once again to be the best, obtaining a score of 51%, far below the average 80% achieved by the other models, with most non-commercial models performing  80%. Sense Polysemy Degree Importance (SPDI) Similarly to SFII, we also study the extent to which the polysemy degree, i.e., how many senses a given word can have, impacts the models' disambiguation capabilities. This experiment mirrors SFII, but groups items by their lemma's polysemy degree  EN (  XP ) instead of  . Figure 3(b) reports the results on all items. Unsurprisingly, similarly to the frequency index, we observe that higher polysemy leads to more errors, confirming that models still struggle with very polysemous words. Similarly to SFII, SPDI is defined as the average percentage of DeepL Google M2M100 M2M100 LG MBart50 MBart50 MTMOPUS Mean MFS MFS+ MFS MFS+ MFS MFS+ MFS MFS+ MFS MFS+ MFS MFS+ MFS MFS+ MFS MFS+ DE 53.68 84.21 56.76 86.82 61.28 87.23 59.13 87.30 58.89 89.72 55.82 89.56 56.98 87.92 57.51 87.54 ES 59.89 87.91 61.96 89.05 61.81 89.37 61.78 88.03 60.17 91.10 63.09 91.85 64.47 91.21 61.88 89.79 IT 68.08 86.38 61.96 87.23 60.75 86.79 62.82 88.81 62.90 87.50 68.97 91.81 64.48 89.66 64.28 88.31 RU 50.00 83.33 48.12 83.28 47.87 83.41 45.25 84.16 47.39 87.20 44.91 87.96 48.40 84.04 47.42 84.77 ZH 49.07 88.89 56.05 88.20 59.06 91.34 59.35 92.45 50.66 89.87 54.17 90.28 51.71 87.45 54.30 89.78 Mean 56.14 86.15 56.97 86.92 58.15 87.63 57.66 88.15 56.00 89.08 57.39 90.29 57.21 88.06 57.08 88.04 Table 5: Frequency Analysis: MFS represents the average percentage of times the model mistakenly translates the target word into a lexicalization belonging to the Most Frequent Sense associated with  P . errors at varying polysemy degrees, and its values are reported in Table 4: once again, DeepL outperforms all other systems by a large margin, confirming that it is the least biased across the board. Most and More Frequent Senses To further corroborate our findings about semantic biases, we study how often models predict senses that are more frequent than the target one. Given a BAD analyzed item XM L , we denote   as the synset associated with the wrongly translated lemma  L . 15 Then, we check the frequency of  and   with respect to  XP : if   XP (  ) <   XP (  ) , then the sys-tem's disambiguation steered towards a sense that is more frequent than the target one, which we 15 In the case in which there are multiple possible synsets, we take the most frequent according to   XP , as we need to rely on the assumption that the surface form represents the intrinsic disambiguation performed by the NMT system. dub More Frequent Sense (MFS+); additionally, if   XP (  ) = 1 , then the model disambiguated the source word w i to the Most Frequent Sense (MFS) of the associated lemma  XP . The results of both these analyses are reported in Table 5. We can observe a few interesting results: first, on average, almost 60% of the time a mistake reflects the Most Frequent Sense of the target word (second-last column); second, almost 90% of the errors concern translations towards more frequent senses of the target word (last column). Importantly, these results are consistent across systems, whether commercial or not. Although it might seem straightforward, NMT models are still strongly biased towards senses that are more likely to be encountered during training; while this could be related to the pattern-matching nature of neural networks, it also depends heavily on the training data the model was trained upon, and this needs to be further investigated in future research. The existing literature in WSD points to the fact that verbs are generally harder than nouns, mostly due to their highly polysemous nature (Barba et al., 2021b). We try to analyze whether MT models 0 10 20 30 40 0 500 1000 1500 0 0.2 0.4 0.6 0.8 1 # i t e m s are affected by the same phenomenon: in Table 6, we report the average results obtained by running DIBIMT on all its sentences (column ALL) and the subset of sentences whose target word was either a NOUN or a VERB. In general, we observe an average drop of accuracy of 4 points, as well as an astounding difference of 18 percentage points in MISS handling, which we will investigate more thoroughly in future work. Interestingly, MT models are much more inclined to translate nouns into their most frequent sense; we attribute this difference to the generally higher polysemy of verbs compared to nouns, which increases the size of the space of possible translations for a given verb, thus decreasing the chance that it gets translated into the MFS. Aside from this, we draw the same conclusion as that drawn by previous works in the field of WSD, with nouns being generally easier to translate than verbs. We try to assess to what extent, in a multilingual encoder-decoder architecture, the encoder is determining the implicit disambiguation of the source sentence before generating the translation. For instance, we ask ourselves this question: given an ambiguous word w i in the source sentence s , how often does the model translate it into a lexicalization representing the same sense, if prompted to translate s into different languages? Intuitively, if the encoder was the sole contributor to the implicit disambiguation performed by the model, we would expect to see the meaning to always be the same, regardless of the target language. and L 2 and an initial item X , we take M 's analyzed items XM L 1 and XM L 2 17 and check if translations in L 1 and L 2 have a synset in common, i.e., |  L 1 (  L 1 )   L 2 (  L 2 ) | > 0 . The results of this experiment are reported in Figure 4. We observe that, on average, this phenomenon occurs around 70% of the time. Hence, it is safe to assume that, while the encoder certainly plays an important role in the disambiguation of the input sentence, the decoder is also contributing sig-nificantly. Another interesting observation is that the alphabet of the target language does not seem languages. We also disregard DeepL and Google Translate as their architecture is proprietary. 17 We skip item X if either XML 1 or XM L 2 is a MISS . to have any influence, as language pairs involving Russian display scores that are very similar to those of the other three European languages. We attribute lower scores in Chinese to coverage issues in BabelNet, which would hinder a correct fulfillment of the condition defined for this experiment. Given the low performances achieved by MT models, we test a WSD system on the English sentences within DIBIMT, both to assess the toughness of our system and to establish an additional baseline. We use ESCHER 18 (Barba et al., 2021a), a state-of-the-art model on English WSD. Interestingly, ESCHER achieves an overall accuracy score of 66.33, almost 15 points lower than the results on the standard WSD benchmark (80.7 on ALL, Raganato et al., 2017), therefore confirming the challenging nature of DIBIMT. Furthermore, in order to estimate the difference in disambiguation capability between NMT models and a dedicated WSD system, we compute ESCHER's performances on the set of English sentences of nonMISS analyzed items for each (model, language) pair. We report these results in Table 7, whose accuracy scores can be directly compared to those in Table 3. As expected, the average MT accuracy is sig-nificantly lower than ESCHER's, with the sole exception of DeepL, which manages to surpass it on German and Russian. These results clearly demonstrate that current NMT models are still not on par with dedicated WSD systems, and thus that they might benefit from the inclusion of such WSD systems within the NMT ecosystem. As a final experiment, we assess whether the semantic biases are caused by search errors (i.e., failures of the decoding algorithm), or model errors (i.e., the models deemed their translations the best 18 The publicly available version trained on SemCor data only. M2M100 M2M100 LG MBart50 MBart50 MTMOPUS Mean DE 98.00 98.00 92.00 94.00 84.00 93.20 ES 100.00 98.00 88.00 90.00 94.00 94.00 IT 94.00 90.00 86.00 100.00 88.00 91.60 RU 94.00 90.00 98.00 92.00 88.00 92.40 ZH 96.00 98.00 94.00 98.00 92.00 95.60 Mean 96.40 94.80 91.60 94.80 89.20 93.36 Table 8: Model Errors: percentage of times a model thought its BAD translation was better than a GOOD one. possible). For each (model M , language L ) pair, we sample a BAD translation ( t BAD ), pair it with a GOOD translation ( t GOOD ) produced by another model (prioritizing DeepL), and ask annotators to check their correctness and apply corrections where needed, 19 then compute the perplexities according to M with the corresponding English sentence, and call them p GOOD and p BAD respectively. We repeat this sampling 50 times per ( M , L ) pair and check how often p BAD < p GOOD . Table 8 shows that, on average, this happens in 93% of cases, thus confirming that most semantic biases are embedded within models and are not caused by the decoding strategy. In this work, we presented DIBIMT, a novel benchmark for measuring and understanding semantic biases in NMT, which goes beyond simple accuracy and provides novel metrics that summarize how biased NMT models are. We tested DIBIMT on 7 widely adopted NMT systems, extensively discussing their performances and providing novel insights into the possible causes and relations of semantic biases within NMT models. Furthermore, statistics of our annotations suggest that, when dealing with translations, synsets' lexicalizations cannot be used interchangeably, as their choice depends heavily on the context. In the future, we plan to improve DIBIMT by introducing better heuristics to recognize and handle MISS cases, especially covering the linguistic phenomena we described (see Section 4.2); we also aim at widening language coverage and increasing the number of sentences in the benchmark, consequently improving word and sense coverage. To enable further research, we release DIBIMT as a closed benchmark with a public leaderboard at: https://nlp.uniroma1.it/dibimt . 19 We do this to make the translations more grammatically fluent, and not to correct the disambiguation of the target term, which was never detected as being wrong in the sampled cases. The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 and the ELEXIS project No. 731015 under the European Union's Horizon 2020 research and innovation programme, and the PerLIR project (Personal Linguistic resources in Information Retrieval) funded by the MIUR Pro-getti di ricerca di Rilevante Interesse Nazionale programme (PRIN 2017). This work was also partially supported by the MIUR under the grant Dipartimenti di eccellenza 2018-2022" of the Department of Computer Science of Sapienza University. References Edoardo Barba, Tommaso Pasini, and Roberto Navigli. 2021a. Esc: Redesigning WSD with extractive sense comprehension. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 46614672. Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2021b. ConSeC: Word Sense Disambiguation as continuous sense comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 14921503, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, Roberto Navigli, et al. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 . International Joint Conference on Artificial Intelligence, Inc. Denis Emelin, Ivan Titov, and Rico Sennrich. 2020. Detecting word sense disambiguation biases in machine translation for model-agnostic adversarial attacks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 76357653, Online. Association for Computational Linguistics. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond English-Centric Multilingual machine translation. Journal of Machine Learning Research , 22(107):148. Annette Rios Gonzales, Laura Mascarell, and Rico Sen-nrich. 2017. Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings. In Proceedings of the Second Conference on Machine Translation , pages 1119. Annette Rios Gonzales, Mathias Mller, and Rico Sen-nrich. 2018. The Word Sense Disambiguation Test Suite at WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers , pages 588596. Robert Krovetz and W Bruce Croft. 1992. Lexical ambiguity and information retrieval. ACM Transactions on Information Systems (TOIS) , 10(2):115 141. Helen Langone, Benjamin R Haskell, and George A Miller. 2004. Annotating WordNet. In Proceedings of the Workshop Frontiers in Corpus Annotation at HLT-NAACL 2004 , pages 6369. Els Lefever and Vronique Hoste. 2013. Semeval-2013 task 10: Cross-lingual Word Sense Disambiguation. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) , pages 158166. Frederick Liu, Han Lu, and Graham Neubig. 2018. Handling Homographs in Neural Machine Translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 13361345, New Orleans, Louisiana. Association for Computational Linguistics. Rebecca Marvin and Philipp Koehn. 2018. Exploring Word Sense Disambiguation Abilities of Neural Machine Translation Systems. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) , pages 125131. Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 31033114, Minneapolis, Minnesota. Association for Computational Linguistics. Rada Mihalcea, Ravi Sinha, and Diana McCarthy. 2010. Semeval-2010 task 2: Cross-Lingual Lexical Substitution. In Proceedings of the 5th international workshop on semantic evaluation , pages 914. George A Miller, Richard Beckwith, Christiane Fell-baum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography , 3(4):235 244. Roberto Navigli. 2009. Word Sense Disambiguation: A Survey. ACM computing surveys (CSUR) , 41(2):169. Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi. 2021. Ten Years of Babelnet: A Survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 45594567. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations . Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pages 99110, Valencia, Spain. Association for Computational Linguistics. Alessandro Raganato, Yves Scherrer, and Jrg Tiedemann. 2019. The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) , pages 470480. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual Translation from De-noising Pre-Training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 34503466, Online. Association for Computational Linguistics. Jrg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT  Building open translation services for the World. In Proceedings of the 22nd Annual Con-ferenec of the European Association for Machine Translation (EAMT) , Lisbon, Portugal. David Vickrey, Luke Biewald, Marc Teyssier, and Daphne Koller. 2005. Word-Sense Disambiguation for Machine Translation. In Proceedings of human language technology conference and conference on empirical methods in natural language processing , pages 771778. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rmi Louf, Morgan Fun-towicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Can-wen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 3845, Online. Association for Computational Linguistics. A Analysis Procedure Details Our analysis procedure, which we described in Section 3.4, involves steps that go beyond simple lemma matching. For instance, in case of multiword expressions, we allowed annotators to specify a wildcard, i.e., any number of tokens (including zero) were allowed to expand and still trigger a match. Additionally, since Stanza has multi-word expansion tokenization for some of the languages in our list, when available, we try to perform matching on both the list of words (alongside the list of tokens) in the translated sentence. Finally, in case no match is produced by the aforementioned steps, we apply a surface-level string matching heuristic which, especially in Chinese, helps us increase coverage. B Neural Models Implementation We use HuggingFace's Transformers library (Wolf et al., 2020) for all neural models. As per standard practice, we generate translations using beam search as decoding algorithm with beam size 5. C Instructions for Dataset Annotation In this work, we investigate semantic biases in Machine Translation across languages. You are provided with a spreadsheet containing 300 instances, each including the following information: a lemma, its part of speech, a definition and some good and bad translation candidates derived from BabelNet. Your task is to manually verify the correctness of the good candidates and add new good candidates if deemed necessary. Furthermore, you are asked to verify that all bad candidates are wrong. From a translation perspective, a good candidate is a word which correctly translates the English target word in the given context. Instead, a bad candidate is a wrong translation of the English target word in the given context. Please adopt the following guidelines while annotating:  Do not annotate idioms.  Do not annotate instances in which the semantic context does not allow us to unequivocally determine the meaning of the target word.  Do not annotate proper names, e.g., Run in the sentence The military campaign near that creek was known as The battle of Bull Run  .
abstain abstain abstain objective objective other abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain objective abstain objective objective objective result other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain other abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain method method method method abstain abstain abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain other abstain abstain abstain objective objective method objective other method other other