With state-of-the-art systems having finally attained estimated human performance, Word Sense Disambiguation (WSD) has now joined the array of Natural Language Processing tasks that have seemingly been solved, thanks to the vast amounts of knowledge encoded into Transformer-based pre-trained language models. And yet, if we look below the surface of raw figures, it is easy to realize that current approaches still make trivial mistakes that a human would never make. In this work, we provide evidence showing why the F1 score metric should not simply be taken at face value and present an exhaustive analysis of the errors that seven of the most representative state-of-the-art systems for English all-words WSD make on traditional evaluation benchmarks. In addition, we produce and release a collection of test sets featuring (a) an amended version of the standard evaluation benchmark that fixes its lexical and semantic inaccuracies, (b) 42D, a challenge set devised to assess the resilience of systems with respect to least frequent word senses and senses not seen at training time, and (c) hardEN, a challenge set made up solely of instances which none of the investigated state-of-the-art systems can solve. We make all of the test sets and model predictions available to the research community at https://github.com/ SapienzaNLP/wsd-hard-benchmark . In recent years, Natural Language Processing (NLP) has witnessed a quantum leap in benchmark task performance, mainly thanks to the adoption of two major technical innovations: the Transformer architecture (Vaswani et al., 2017) and transfer learning from language models pre-trained on massive amounts of textual data (Devlin et al., 2019; Lewis et al., 2020). The impact of these breakthroughs was so strong that, on many benchmarks, the performance of human non-experts was surpassed (Wang et al., 2019b), prompting researchers to release new, more challenging benchmarks (Wang et al., 2019a). Word Sense Disambiguation (WSD), the task of automatically assigning a meaning to an ambiguous word in context (Bevilacqua et al., 2021), is undergoing a similar process: current state-of-the-art systems are now capable of attaining and surpassing the F1 score of 80% 1 on standard test datasets (Bevilacqua and Navigli, 2020; Barba et al., 2021a; Conia and Navigli, 2021; Kohli, 2021), a figure often reported as the estimated human performance, because it corresponds to the highest recorded inter-annotator agreement (Edmonds and Kilgarriff, 2002; Navigli et al., 2007; Palmer et al., 2007). Matching and/or surpassing human performance reasonably triggers the assumption that systems are capable of carrying out tasks in real-world scenarios as effectively as their human counterparts (Kiela et al., 2021), to the point where non-practitioners would regard such tasks as solved. And yet, once systems are investigated beyond sheer accuracy figures, their flaws become readily apparent (Ribeiro et al., 2016; Belinkov and Bisk, 2018; Ribeiro et al., 2020; Card et al., 2020; Zhou et al., 2020). Following this trend of research, our work provides evidence showing why traditional evaluation measures for WSD, such as the F1 score, should not be taken at face value, hence corroborating the thesis that the problem of disambiguation is far from solved (Emelin et al., 2020; Loureiro et al., 2021). To provide context, consider the following example, where the sense prediction 2 of the currently state-of-the-art ESCHER model (Barba et al., 2021a) for the word wind is compared with the gold answer from the test set of SemEval-2013 Task 12 (Navigli et al., 2013): 1 Unless specified, for the remainder of this work, we will use F1 score to refer to the micro-averaged F1 score. 2 According to the most commonly employed sense inventory for WSD, i.e., WordNet 3.0 (Fellbaum, 1998). context: The banks battling against a strong wind in the USA several years later. Investors and regulators (. . . ) gold: A tendency or force that influences events. ESCHER: Air moving (. . . ) from an area of high pressure to an area of low pressure. Here, the contextual meaning of the word wind is clear to any English speaker, with no cues in the sentence that would lead a human reader to pick the air meaning. This is an illustrative case of why, despite having achieved (on paper) superhuman performance, systems continue to make mistakes that the inter-annotator agreement would not justify. Similarly, in the context below, another system which breaks the 80% performance ceiling (Conia and Navigli, 2021) makes a trivial mistake on a standard test instance (Snyder and Palmer, 2004), and fails to label the word couple properly: context: I was just sitting down to meet with some new therapy clients, a couple, and the building started shaking (. . . ) gold: A pair of people who live together. With a view to gaining a better understanding of the nature of what systems still fail to disambiguate, in this work we provide the following main contributions: (i) we put forward a detailed quantitative and qualitative analysis of errors shared among seven state-of-the-art systems for English WSD, including systems that have surpassed the 80 % human estimate in terms of F1 score (Bevilacqua and Navigli, 2020; Barba et al., 2021a), (ii) we produce an amended version of the English all-words WSD evaluation benchmarks featured in Senseval and SemEval tasks (Agirre et al., 2009; Raganato et al., 2017a), (iii) we devise 42D (pron. [fortitude]), the first manually-curated test bed made available to the research community after a hiatus of seven years since SemEval-2015 (Moro and Navigli, 2015), and a powerful evaluation tool for estimating system resilience in contexts featuring least frequent word senses, (iv) we establish a new human performance threshold for assessing actual superhuman scores on WSD test sets, and propose macro-averaged F1 score as an alternative to micro-averaged F1 score to better account for least frequent word senses in WSD evaluation, (v) we release hardEN, a challenge set for English all-words WSD on which state-of-the-art systems under investigation achieve exactly 0.0% F1 score, and (vi) we set up an experimental setting to show the impact sense distribution has over the aforementioned datasets. WSD has witnessed the creation of many different evaluation benchmarks, most notably as part of the Senseval (now SemEval) evaluation campaigns (Kilgarriff, 1998). Since the release of the popular Unified Evaluation Framework by Raganato et al. (2017a), the experimental setting has become quite standard, with most systems being evaluated on ALL, i.e., the concatenation of Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 Task 1 (Snyder and Palmer, 2004), SemEval-2007 Task 17 (Pradhan et al., 2007), SemEval-2013 Task 12 (Nav-igli et al., 2013), and SemEval-2015 Task 13 (Moro and Navigli, 2015). Besides reporting results split by part of speech, which has not been particularly insightful, no specific finer-grained analysis is usually performed. 3 This trend runs the risk of promoting a sort of collective hill-climbing behavior, which, in turn, makes it unclear how much the improvement in performance has been due to genuinely stronger generalization power, as opposed to overfitting to increasingly stale test sets. In opposition to this measure-centered style of evaluation, one possible alternative is that of behavioral testing, as proposed by Ribeiro et al. (2020). In their proposal (which does not address WSD ex-plicitly), the benchmark evaluates separately minimum testable units of behavior, each of which addresses one specific skill required by a usable system. WSD, however, is a tricky problem to address in this way, as it is, in fact, a collection of idiosyncratic, diverse classification problems, which are hard to cluster in a meaningful way. A different kind of analysis, perhaps more specific to WSD, has tackled the problem of the strong imbalance of sense distributions, which makes learning difficult for automatic algorithms, and monitors how this imbalance affects performance (Calvo and Gelbukh, 2015; Izquierdo et al., 2015; Postma et al., 2016; Wang and Wang, 2021). We 3 Partial exceptions are Kumar et al. (2019), Bevilacqua et al. (2020), Blevins et al. (2021), Chen et al. (2021), and Barba et al. (2021a), which have paid particular attention to least frequent senses and data efficiency. follow this line of research in that we also take sense distribution skewness as the core issue in the development of WSD algorithms. Therefore, both in the analysis of current WSD systems and in the creation of our new benchmarks, we check for the excessive influence of the most frequent output classes. In an effort to make our analysis as thorough and comprehensive as possible, we consider a set of seven representative cutting-edge approaches for WSD. 4 With the exception of SyntagRank (Scoz-zafava et al., 2020), all systems are supervised neural architectures exploiting pre-trained language models. Below, we describe each of these systems: 5 ARES (Scarlini et al., 2020) 6 is a semi-supervised approach to producing contextualized sense embeddings that share the same space as those from BERT (Devlin et al., 2019). It enables a simple 1 Nearest-Neighbour algorithm to attain high performance both in the English and multilingual settings despite relying on English training data only. We use the ARES English vectors freely available at http://sensembert.org . BEM (Blevins and Zettlemoyer, 2020) is a bi-encoder model with high accuracy for the disambiguation of rare word senses. BEM maps the target in context and its word senses (as represented by glosses) independently into a shared embedding space, by means of jointly learned context and gloss encoders. Disambiguation is then performed simply by predicting the sense whose encoding is most similar to that of the target. We employ the model and code available at https://github.com/ facebookresearch/wsd-biencoders . ESCHER (Barba et al., 2021a, ESR) frames WSD as a span extraction task similar to SQuAD (Rajpurkar et al., 2016), in which a system is asked to detect the span matching the gloss of the correct sense for a target word from a pseudo-document constructed by concatenating the con-4 To ensure a fair comparison, we only consider sys-tems/settings that are not exposed to the Princeton WordNet Gloss Corpus ( https://wordnetcode.princeton. edu/glosstag.shtml ). text of the target word with all the glosses of its possible senses. At the time of writing, ESCHER represents the state of the art in WSD. 7 We employ the model and code available at https: //github.com/SapienzaNLP/esc . EWISER (Bevilacqua and Navigli, 2020, EWR) is a WSD classifier that exploits relational information included in WordNet by incorporating a sparse adjacency matrix within the architecture. We employ the model and code available at https: //github.com/SapienzaNLP/ewiser . Generationary (Bevilacqua et al., 2020, GEN) reframes WSD as definition modeling, i.e., the task of generating a gloss from static or contextual embeddings (Noraset et al., 2017), therefore recasting disambiguation as a generative problem. We use the GEN-UNI (MBRR) model reported in the original paper. While the only exposure of the model to WordNet-tagged data was through SemCor (Miller et al., 1993), i.e., the most widely employed training set for WSD, the model was also trained on other lexicographic resources, such as the Oxford Dictionary (Chang and Chen, 2019). GlossBERT (Huang et al., 2019, GLB) formulates WSD as a gloss ranking task, with a cross-encoder scoring context-gloss pairs. The model is trained with a simple learning-to-rank (He et al., 2008) approach, simply predicting whether a gloss is relevant to the context or not. We employ the model and code available at https://github. com/HSLCY/GlossBERT . SyntagRank (Scozzafava et al., 2020, SYN) is a knowledge-based system that jointly exploits the Personalized PageRank algorithm and the wealth of syntagmatic information contained in SyntagNet (Maru et al., 2019) to perform disambiguation in multiple languages. We accessed SyntagRank by means of its APIs which are freely available at http://api.syntagnet.org/ . To consider WSD as solved, it would be reasonable to expect disambiguation errors to be little more than mismatches between the reference ground truth and another different, but still reasonable interpretation. For example, if we consider the word 7 Contemporary to this work, ConSeC (Barba et al., 2021c), which extends ESCHER, has now attained the new state of the art. 4726 dataset #inst #mono ARES BEM ESR EWR GEN GLB SYN gold ALL 7,253 1,301 71.3% 72.6% 71.2% 72.7% 69.0% 74.8% 81.1% 65.2% ALLHC 541 0 64.7% 71.0% 68.6% 67.8% 62.7% 70.6% 80.2% 2.0% ALL 7,253 1,301 88.2% 87.4% 86.3% 88.8% 85.9% 88.6% 88.8% 84.3% ALLHC 541 0 96.9% 96.7% 96.5% 98.0% 95.0% 97.2% 98.3% 67.1% Table 1: Times (%) systems predict the MFS in WordNet, i.e., WN1st (top), or a sense occurring at least once in SemCor (bottom). chestnuts in my aunt grows chestnuts, the two senses any of several attractive deciduous trees yellow-brown in autumn and edible nut of any of various chestnut trees of the genus Castanea would both be good, albeit slightly different interpretations, but the sense the brown color of chestnuts, instead, is clearly not. To show that the current state of the art is nowhere near this level of performance, we select as a case study the set of instances in the Unified Evaluation Framework for English WSD of Raganato et al. (2017a) (ALL) which are wrongly disambiguated by all of the considered systems (see Section 3). We analyze this hard core (henceforth, ALLHC )where performances are 0 . 0 % in F1 score across the board by designfrom both a quantitative and a qualitative perspective. Sense distribution is a central problem for WSD. In our quantitative study, therefore, we analyze performances on the hard core by dividing test instances into frequency-based partitions. While performances are virtually always computed in terms of micro-averaged F1 scores, here we choose to report macro-averaged F1 (aggregated by sense), as the former gives more weight to frequent senses simply because they occur more oftenthus hiding mediocre performances on least frequent senses. Most Frequent Sense Bias. The most frequent class (in WSD, the most frequent sense, or MFS) can be overpredicted by machine learning algorithms (Postma et al., 2016; Blevins and Zettle-moyer, 2020; Loureiro et al., 2021). To quantify this phenomenon, in Table 1 (top), we report how many times our systems at issue predict the MFS in WordNet (henceforth, WN1st) on ALLHC , as well as on ALL itself. 8 As can be seen, systems show a clear bias towards WN1st senses on ALL, predicting them much more often (at least 69% ) than the WN1st rate on the ground truth ( 65 . 2% ). The distribution divergence becomes dramatic on ALLHC , where systems predict WN1st at least 62 . 7% of the times, but the true WN1st rate is now just 2 . 0% . Overall, systems show a mostly comparable bias towards WN1st, with two notable exceptions: (i) GEN, likely due to the fact that in its UNI setting the system is exposed to multiple resources and hence is less biased; on the other hand, and perhaps coun-terintuitively (but see Calvo and Gelbukh, 2015) (ii) SYN, which is unsupervised, is the most biased 8 We consider a test set instance to be a WN1st instance if at least one of the word senses assigned to disambiguate it coincides with the WN1st. towards WN1st. Finally, we note that ESR, despite being the state of the art, does not behave differently from other systems in this respect, suggesting that there is much room for improvement. In Table 2, we report both microand macro-averaged F1 scores on ALL, a subset of ALL without WN1st instances (ALL no 1 st ), and ALLHC . As a consequence of the reduced importance of frequent senses, macro-averaged F1 scores are always lower than micro-averaged counterparts. Moreover, we can see that the reduced bias on WN1st by GEN results in a partial divergence between the system ranking on ALL and that on ALL no 1 st , with GEN, which has a much lower WN1st bias, now outperforming GLB on the latter. Training Dataset Bias. In addition to the WN1st bias, it is also useful to examine how much the lack of extrapolative capabilities is a reason for the existence of such a large set of unanswerable items. Thus, we classify instances and predictions according to whether the sense occurs at least once in SemCor (see also Kumar et al., 2019; Wang and Wang, 2021). Predicting a sense that never occurs at training time not only requires zero-shot capabilities, but also the ability to overcome the bias that a system learns from the training data for other senses of the same word. In Table 1 (bottom), we report the frequency with which our systems at issue predict a word sense that occurs at least once in SemCor. If we look at the raw percentages for ALL, there seems to be a slight bias towards senses that were seen at training time. However, such values do not take into account monosemous words for which the model always outputs the correct answer. In ALLHC , where by construction there cannot be any monosemous sense, occurring senses are predicted at least 95% of the times, while they make up only 67 . 1% of the ground truth. We refer back to Table 2 for the F1 scores on ALL noSC , i.e., the subset of ALL with no instances whose gold sense is found in SemCor. The divergence between the ranking on ALL and ALL noSC is even wider than that between ALL and ALL no 1 st . In this case, GEN, which obtains rather unremarkable results on ALL, becomes the second-to-best on ALL noSC , supporting the notion that gloss modeling is beneficial for WordNet-based WSD, even when using data outside of WordNet. Indeed, the gloss-centric approach of ESR offers the best results across the board, even though its bias on SemCor-attested (and WN1st) senses is still stronghinting that a possible way forward could be combining ESR (or any equally strong baseline) with strategies meant to mitigate the bias. Determining why a sizeable subset of instances cannot be disambiguated by any of the systems we take into consideration requires a finer-grained, qualitative level of analysis to check whether, i) annotation errors, or ii) gaps in WordNet, are an important factor. At the same time, iii) we also want to see if we replicate previous inter-annotator agreement figures (Edmonds and Kilgarriff, 2002; Navigli et al., 2007; Palmer et al., 2007). In order to achieve these objectives, we ask an expert linguist with extensive experience in tagging with the WordNet inventory 9 to revise the test instances in ALL, the main test set first provided by Raganato et al. (2017a), 10 as well as in the dataset released as part of SemEval-2010 in-domain WSD Task 17 of Agirre et al. (2009), by tagging each instance with one of the following labels:  unchanged , to indicate that the annotator agreed with the existing ground truth;  fine-grained , to indicate that one or more senses need to be added to the ground truth, without removing the existing ones;  error:token-lemma , to indicate that the test instance was originally assigned a wrong lemma, or was improperly tokenized;  error:pos , to indicate that the test instance was originally assigned a wrong part of speech (PoS);  error:sense , to indicate that one or more senses in the ground truth are wrong;  error:inventory , to indicate that the ground truth is wrong, but there is no appropriate sense for the target word in the inventory of WordNet 3.0. Table 3 showcases an excerpt of instances as tagged by our linguist according to the aforementioned set of labels. Additionally, in Table 4, we 9 All our annotators have effective operational proficiency in English and received a wage in line with their country of residence. Annotation was carried out by means of a user-friendly, in-house interface. 10 We exclude SemEval-2007, since this dataset is often used as development set (Pasini et al., 2021). provide a broader look and report the frequency of appearance (percentage) for each label, as assigned to (a) the concatenation of datasets in Raganato et al. (2017a) with the exception of monosemous words and SemEval-2007 instances (ALL-), (b) its subset of shared errors making up the hard core described in Section 4 (ALLHC-), (c) ALLnot including instances featured in ALLHC(ALLNS-), and (d) SemEval-2010 with no monosemous instances (S10-). Two interesting results emerge from this analysis. On the one hand, the hard core seems to be hard for the human annotator too, since the majority of instances are labeled as either disambiguation errors ( error:sense ), or as lacking equally valid word senses ( fine-grained) . Indeed, the shared error subset (ALLHC-) features the lowest level of unchanged instances and, at the same time, the highest rate of error:sense instances, meaning that the linguist had a significantly higher disagreement with respect to the original test set in ALLHCthan in ALLNS-. Furthermore, the percentage of cases in which the linguist deemed necessary the use of (i) additional word senses to disambiguate a certain instance ( fine-grained ) or (ii) the use of a word sense not featured in the inventory ( error:inventory ) is more than double that of the rest of the dataset. On the other hand, if we sum the percentage of unchanged instances with that of fine-grained , and exclude from the set of all instances the samples where disagreements do not depend on disambiguation choices ( error:pos , error:token-lemma , error:inventory ), the agreement of the linguist with respect to the gold standard is far superior to what is traditionally reported in the literature, reaching a high ceiling of 91 . 1% , more than 10% above traditional estimates (Ed-monds and Kilgarriff, 2002; Navigli et al., 2007; 4729 Palmer et al., 2007). Indeed, fine-grained instances do not involve a disambiguation error, but merely extend the instance with additional possible meanings. This can only increase performances, since the standard evaluation scorer provided as part of the framework of Raganato et al. (2017a) gives the system full score if the predicted sense is in the ground truth set. Results from the quantitative and qualitative analysis carried out on the hard core reveal two main reasons why F1 scores can be potentially misleading indicators of the actual capabilities of current systems: (i) scores are actually a long way from estimated human performance when observed in challenging, but nevertheless real-world scenarios, and (ii) errors found in traditional test beds compromise insightful model evaluations. Against this background, we put forward a set of evaluation tools to enable a more robust appraisal of system performance in English WSD, namely, (i) 42D, a multi-domain challenge set, (ii) amended versions of ALL (ALLNEW ) and SemEval-2010 Task 17 (S10 NEW ), and (iii) the new hardEN/softEN benchmark. Thus far, we have only considered existing evaluation benchmarks for WSD. In view of thisand with the purpose of showing that the issues highlighted in Section 4.1 are not artifacts of the data taken into account, but a general problem with current WSD systemswe introduce 42D, a novel test set for English WSD, built from scratch by manually annotating paragraphs taken from the British National Corpus (Leech, 1992, BNC). 11 42D, with its 370 test instances, is specifically designed to be a challenge set (Belinkov and Glass, 2019), since for each of the instances the ground truth, i) does not occur in SemCor, and ii) is not the first sense in WordNet. In addition to this, 42D's source texts are sampled so as to be representative of different text domains, specifically, the 42 domains defined in BabelNet 12 4.0 (Navigli and Ponzetto, 2012; Nav-11 This work was endorsed by the BNC staff via the official inquiry mail ( ota@bodleian.ox.ac.uk ) on October 15 , 2019 and it complies with the BNC Licence for the use of paragraphs and other fragments ( http://www.natcorp. ox.ac.uk/faq.xml?ID=licensing ). 12 BabelNet is freely available for research purposes at https://babelnet.org . igli et al., 2021). 13 5.2 ALLNEW and S10 NEW With the aim of providing a cleaner test set, one in which non-system-dependent issues have been removed, we ask the same linguist who performed the error analysis of Section 4.2 to complete the task by also updating the instances from ALL and SemEval-2010 based on the labels assigned during the first phase: additional word senses are assigned for instances labeled as fine-grained and existing annotations are amended for error:sense cases; PoS tagging, lemmatization, and tokenization errors are fixed, and the instance updated with suitable word senses (see Table 3 for an excerpt of changes applied to the original test sets). As a result, we obtain two test sets: ALLNEW , featuring 4 , 917 polysemous instances amending the original ALL dataset of Raganato et al. (2017a) 14 , and S10 NEW , with 955 polysemous test instances amending the original SemEval-2010 Task 17 of Agirre et al. (2010). Besides an analysis of the current WSD evaluation datasets, in this paper we also want to make available one comfortable-to-use benchmark that addresses the discussed issues. For this reason, we derive a new intersection of 476 test instances that the systems at issue were not able to solve, this time, from the concatenation of the amended sets ALLNEW and S10 NEW , as well as 42D. We name this challenge set hardEN, in contrast to its counterpart, softEN, which, instead, features the remaining 5 , 766 test instances for which at least one system is able to provide a correct prediction. The hardEN/softEN benchmark is useful in that it sets a new starting line for WSD systems, one that concurrently accounts for what they still fail to do, while keeping track of what they can already do. Table 5 compares the results obtained on our revised ALLNEW dataset by the current state-of-the-art systems, with respect to the original ALL test set of Raganato et al. (2017a)filtered to include only instances featured in ALLNEW (ALL  ), showing that the ranking of the systems taken into account does not change as a result of the amending process. However, we can appreciate the significant difference in terms of performance when this is measured using the macro-averaged F1 score as opposed to the micro-averaged F1 score used in the literature. For example, the performance of ESCHER drops by almost 3 points on ALLNEW , from 81 . 6% to 78 . 7% . Indeed, the macro-averaged F1 score is better suited to highlighting the weaknesses of a system with imbalanced class distributions, as is the case for word senses, whose distribution follows Zipf's Law. We argue, therefore, that future systems should also report their results using this measure in order to better enable their strengths and weaknesses to be determined. Table 5 also shows the performance of each system on our revised SemEval-2010 (S10 NEW ), 42D, and the hardEN/softEN benchmark. 42D is of particular interest as it showcases how the state of the art still struggles in challenging settings, including rare word senses and out-of-domain instances: the best system, ESR, only manages to score 54 . 1% in micro F1, a value that is very distant from the 80% figure originally estimated for human experts. As a last remark, it is worth noting how the performances on softEN for EWR and ESR reach and surpass the threshold of 85% , hence showing figures closer to the new, higher human performance ceiling we described in Section 4.2. In this work, we dived deep into what the current state of the art in WSD can achieve and what the main roadblocks to overcome in the future are. With hardEN as the new frontier to surpass and softEN as a milestone to preserve, in this Section, dataset ESCHER Uniform E. Ranked E. M-F1 m-F1 M-F1 m-F1 M-F1 m-F1 ALLNEW 78.7 81.6 77.8 81.6 78.8 82.3 S10 NEW 78.0 82.1 79.5 83.7 80.7 84.9 42D 58.9 54.1 50.9 46.8 53.2 48.9 softEN 83.7 86.8 82.7 87.6 83.4 88.3 hardEN 0.0 0.0 0.0 0.0 0.0 0.0 Table 6: Macro(M-F1) and micro-averaged F1 (m-F1) scores of our Uniform and Ranked ensemble strategies compared against the best performing systems, ESCHER. Joining forces. One might wonder whether putting together multiple systems can be a viable approach for achieving progress in WSD, as preliminarily explored in the past by (Brody et al., 2006). Here we provide a provisional answer by investigating two simple ensemble strategies with the aim of understanding if it is possible to improve the results by making different and diverse systems agree. In the first ensemble strategy, i.e., uniform ensemble, we apply majority voting among the predictions of each of the seven systems; in the second strategy, i.e., ranked ensemble, each voting system is ranked according to its performance rank on ALLNEW , e.g., the vote of ESCHER (the best system on ALLNEW ) is worth seven times that of SyntagRank (the seventh and worst system), in order to favor systems that are more likely to predict correct senses. sults for ALLNEW are slightly higher when using ranked ensembling, this strategy appears to be impairing performance in challenging settings such as 42D. Furthermore, by construction, if hardEN features all and only those instances that all the systems at issue fail to provide a correct answer for, then ensembles cannot represent a solution for hardEN, no matter the strategy employed. Data augmentation. A renowned problem in WSD is the knowledge acquisition bottleneck: we have thousands of senses for which we have no available training data, but manual sense tagging is an expensive process (Pasini, 2020). What happens when a system is trained with automatically generated usage examples? To find out, we employ the examples generated via the EXMAKER encoder-decoder architecture (Barba et al., 2021b), to train ESCHER in two configurations: the first, in which the system is trained only with one automatically generated example per sense (K1), and the second, in which ESCHER is trained on the concatenation of SemCor and K1 (SemCor+K1). As shown in Table 7, although ESCHER, when using K1, successfully nibbles at hardEN (achiev-ing 35 . 3 % in terms of macro-averaged F1 score), it does so at the expense of its performance on softEN (dropping more than 18 % in macro-averaged F1 score), which is clearly undesirable. This is further proof that flattening the sense distribution on the training set is not sufficient to deal with hard test instances while at the same time preserving performance on the easier ones (see also Postma et al. (2016) and Loureiro et al. (2021)). Although traditional metrics indicate that WSD systems have attained human-level performances, the actual capabilities of state-of-the-art models are poorly reflected by the current evaluation benchmarks. In this paper, we analyzed the intersection of errors made by a heterogeneous set of seven state-of-the-art systems for English WSD from a quantitative and qualitative perspective, detailing two main reasons why they still falter when compared to their human counterparts, namely, their strong bias towards most frequent word senses and towards senses featured in the training data, as well as the presence of an array of lexical and semantic fallacies in traditional evaluation benchmarks. With the aim of providing a test bench that is more effective in reflecting the actual capabilities of WSD systems, we introduced (i) an amended version of the most popular test bed for WSD, and (ii) the 42D challenge set. As a result of the aforementioned work, we also present the hardEN/softEN benchmark, a unified test bed aimed at moving forward with the disambiguation of so far unresolved instances, while keeping track of the current strong points of WSD systems. We make our test sets and model predictions available at https://github.com/ SapienzaNLP/wsd-hard-benchmark . The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487, the ELEXIS project No. 731015 under the European Union's Horizon 2020 research and innovation programme, and the European Language Grid project No. 825627 (Universal Semantic Annotator, USeA). This work was partially supported by the COST Action CA18209 NexusLinguarum European network for Web-centred linguistic data science.
abstain abstain result method abstain abstain abstain other abstain abstain other abstain abstain result method abstain objective objective objective abstain abstain other objective result method abstain objective result result other other other other other other other other abstain method objective method abstain method abstain other abstain abstain abstain other other abstain abstain other abstain other abstain method abstain abstain abstain other abstain abstain other abstain other abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain objective other other other