By design, word embeddings are unable to model the dynamic nature of words' semantics, i.e., the property of words to correspond to potentially different meanings. To address this limitation, dozens of specialized meaning representation techniques such as sense or contextualized embeddings have been proposed. However, despite the popularity of research on this topic, very few evaluation benchmarks exist that specifically focus on the dynamic semantics of words. In this paper we show that existing models have surpassed the performance ceiling of the standard evaluation dataset for the purpose, i.e., Stanford Contextual Word Similarity, and highlight its shortcomings. To address the lack of a suitable benchmark, we put forward a large-scale Word in Context dataset, called WiC, based on annotations curated by experts, for generic evaluation of context-sensitive representations. WiC is released in https://pilehvar.github.io/wic/. One of the main limitations of mainstream word embeddings lies in their static nature, i.e., a word is associated with the same embedding, independently from the context in which it appears. Therefore, these embeddings are unable to reflect the dynamic nature of ambiguous words 1 , in that they can correspond to different (potentially unrelated) meanings depending on their usage in context (Camacho-Collados and Pilehvar, 2018). To get around this limitation dozens of proposals have been put forward, mainly in two categories: multi-prototype embeddings (Reisinger and Mooney, 2010; Neelakantan et al., 2014; Pelevina et al., 2016), which usually leverage context clustering in order to learn distinct representations for individual meanings of words, and contextualized 1 Ambiguous words are important as they constitute the most frequent words in a natural language (Zipf, 1949). word embeddings (Melamud et al., 2016; Peters et al., 2018), which instead compute a single dynamic embedding for a given word which can adapt itself to arbitrary contexts for the word. Despite the popularity of research on these specialised embeddings, very few benchmarks exist for their evaluation. Most works in this do-main either perform evaluations on word similarity datasets (in which words are presented in isolation; hence, they are not suitable for verifying the dynamic nature of word semantics) or carry out impact analysis in downstream NLP applications (usually, by taking word embeddings as baseline). Despite providing a suitable means of verifying the effectiveness of the embeddings, the downstream evaluation cannot replace generic evaluations as it is difficult to isolate the impact of embeddings from many other factors involved, including the algorithmic configuration and parameter setting of the system. To our knowledge, the Stanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) is the only existing benchmark that specifically focuses on the dynamic nature of word semantics. 2 In Section 4 we will explain the limitations of this dataset for the evaluation of recent work in the literature. In this paper we propose WiC, a novel dataset that provides a high-quality benchmark for the evaluation of context-sensitive word embeddings. WiC provides multiple interesting characteristics: (1) it is suitable for evaluating a wide range of techniques, including contextualized word and sense representation and word sense disambiguation; (2) it is framed as a binary classification dataset, in which, unlike SCWS, identical words are paired with each other (in different con-2 With a similar goal in mind but focused on hypernymy, Vyas and Carpuat (2017) developed a benchmark to assess the capability of automatic systems to detect hypernymy relations in context. texts); hence, a context-insensitive word embedding model would perform similarly to a random baseline; and (3) it is constructed using high quality annotations curated by experts. We frame the task as binary classification. Each instance in WiC has a target word w , either a verb or a noun, for which two contexts, c 1 and c 2 , are provided. Each of these contexts triggers a specific meaning of w . The task is to identify if the occurrences of w in c 1 and c 2 correspond to the same meaning or not. Table 1 lists some examples from the dataset. In what follows in this section, we describe the construction procedure of the dataset. Contextual sentences in WiC were extracted from example usages provided for words in three lexical resources: (1) WordNet (Fellbaum, 1998), the standard English lexicographic resource; (2) VerbNet (Kipper-Schuler, 2005), the largest domain-independent verb-based resource; and (3) Wiktionary 3 , a large collaborative-constructed online dictionary. We used WordNet as our core resource, exploiting BabelNet's mappings (Navigli and Ponzetto, 2012) as a bridge between Wiktionary and VerbNet to WordNet. Lexicographer examples constitute a reliable base for the construction of the dataset, as they are curated in a way to be clearly distinguishable across different senses of a word. As explained above, the dataset is composed of instances, each of which contain a target word and two examples containing the target word. An instance can be either positive or negative, depending on whether the corresponding c 1 and c 2 are listed for the same sense of w in the target resource. In order to compile the dataset, we first 3 https://www.wiktionary.org/ obtained all the possible positive and negative instances from all resources, with the only condition of the surface word form occurring in both c 1 and c 2 . 4 The total number of initial examples extracted from all resources at this stage were 23,949, 10,564 and 636 for WordNet, Wiktionary and VerbNet, respectively. We first compiled the test and development sets with two constraints: (1) not having more than three instances for the same target word, and (2) not having repeated contextual sentences across instances. These constraints were enforced to have a diverse and balanced set which covers as many unique words as possible. With all these constraints in mind, we set apart 1,600 and 800 instances for the test and development sets, respectively. We ensured that all the splits were balanced for their positive and negative examples. The remaining instances whose examples did not overlap with test and development formed our initial training dataset. Semi-automatic check. Even though very few in number, all resources (even exprt-based ones) contain errors such as incorrect part-of-speech tags or ill-formed examples. Moreover, the extraction of examples and the mappings across resources were not always accurate. In order to have as few resource-specific and mapping errors as possible, all training, development and test sets were semi-automatically post-processed, either with small fixes whenever possible or by removing problematic instances otherwise. WordNet is known to be a fine-grained resource (Navigli, 2006). Often, different senses of the same word are hardly distinguishable from one another even for humans. For example, more than 40 senses are listed for the verb run , with many of them corresponding to similar concepts, e.g., move fast, travel rapidly, and run with the ball. In order to avoid this high-granularity, we performed an automatic pruning of the resource, removing instances with subtle sense distinctions. Sense clustering is not a very well-defined problem (McCarthy et al., 2016) and there are different strategies to perform this sense distinction (Snow et al., 2007; Pilehvar et al., 2013; Mancini et al., 2017). We adopted a simple strategy and 4 Given that WordNet provides examples for synsets (rather than word senses), a target word (sense) might not occur in all the examples of its corresponding synset. removed all pairs whose senses were first degree connections in the WordNet semantic graph, including sister senses, and those which belonged to the same supersense, i.e. sense clusters from the Wordnet lexicographer files 5 . There are a total of 44 supersenses in WordNet, comprising semantic categories such as shape , substance or event . This coarsening of the WordNet sense inventory has been shown particularly useful in downstream applications (Rud et al., 2011; Severyn et al., 2013; Flekova and Gurevych, 2016; Pilehvar et al., 2017). In the next section we show that the pruning resulted in a significant boost in the clarity of the dataset. To verify the quality and the difficulty of the dataset and to estimate the human-level performance upperbound, we randomly sampled four sets of 100 instances from the test set, with an overlap of 50 instances between two of the annotators. Each set was assigned to an annotator who was asked to label each instance based on whether they thought the two occurrences of the word referred to the same meaning or not. 6 The annotators were not provided with knowledge from any external lexical resource (such as WordNet). Specifically, the number of senses and the sense distinctions of the word (in the target sense inventory) were unknown to the annotators. We found the average human accuracy on the dataset to be 80.0% (individual scores of 79%, 79%, 80% and 82%). We take this as an estimation of the human-level performance upperbound of the dataset. For the overlapping section, we computed the agreement between the two annotators to be 80%. Note that the annotators were not provided with sense distinctions to resemble the more difficult scenario for unsupervised models (which do not benefit from sense-based knowledge resources). Having access to sense defini-tions/distinctions would have substantially raised the performance bar. Impact of pruning. To check the effectiveness of our pruning strategy, we also sampled a set of 100 instances from the batch of instances that were pruned from the dataset. Similarly, the annotators 5 wordnet.princeton.edu/documentation/lexnames5wn 6 Annotators were not lexicographers. To make the task more understandable, they were asked if in their opinion the two words would belong to the same dictionary entry or not. were asked to independently label instances in the set. We computed the average accuracy on this set to be 57% (56% and 58%), which is substantially lower than that for the final pruned set (i.e. 80%). This indicates the success of our pruning strategy in improving the semantic clarity of the dataset. Table 2 shows the statistics of the different splits of WiC. The test set contains a large number of unique target words (1,256), reflecting the variety of the dataset. The large training split of 5,428 instances makes the dataset suitable for various supervised algorithms, including deep learning models. Only 36% of the target words in the test split overlap with those in the training, with no overlap of contextual sentences across the splits. This makes WiC extremely challenging for systems that heavily rely on pattern matching. We experimented with recent multi-prototype and contextualized word embedding techniques. Evaluation of other embedding models as well as word sense disambiguation systems is left for future work. Contextualized word embeddings. One of the pioneering contextualized word embedding models is Context2Vec (Melamud et al., 2016), which computes the embedding for a word in context using a multi-layer perceptron which is built on top of a bidirectional LSTM (Hochreiter and Schmid-huber, 1997) language model. We used the 600d UkWac pre-trained models 7 . ELMo (Peters et al., 2018) is a character-based model which learns dynamic word embeddings that can change depending on the context. ELMo embeddings are essentially the internal states of a deep LSTM-based language model, pre-trained on a large text corpus. We used the 1024d pre-trained models 8 for two configurations: ELMo 1 , the first LSTM hidden state, and ELMo 3 , the weighted sum of the 7 https://github.com/orenmel/context2vec 8 https://www.tensorflow.org/hub/modules/google/elmo/1 3 layers of LSTM. A more recent contextualized model is BERT (Devlin et al., 2019). The technique is built upon earlier contextual representations, including ELMo, but differs in the fact that, unlike those models which are mainly unidirectional, BERT is bidirectional, i.e., it considers contexts on both sides of the target word during representation. We experimented with two pre-trained BERT models: base (768 dimensions, 12 layer, 110M parameters) and large (1024 dimensions, 24 layer, 340M parameters). 9 Around 22% of the pairs in the test set had at least one of their target words not covered by these models. For such out-of-vocabulary cases, we used BERT's default tokenizer to split the unknown word to subwords and computed its embedding as the centroid of the corresponding subwords' embeddings. Multi-prototype embeddings. We experiment with three recent techniques that release 300d pre-trained multi-prototype embeddings 10 . JBT 11 (Pelevina et al., 2016) induces different senses by clustering graphs constructed using word embeddings and computes embedding for each cluster (sense). DeConf 12 (Pilehvar and Collier, 2016) exploits the knowledge encoded in WordNet. For each sense, it extracts from the resource the set of semantically related words, called sense biasing words, which are in turn used to compute the sense embedding. SW2V 13 (Mancini et al., 2017) is an extension of Word2Vec (Mikolov et al., 2013a) for jointly learning word and sense embeddings, producing a shared vector space of words and senses as a result. For these three methods we follow the disambiguation strategy suggested by Pelevina et al. (2016): for each example we retrieve the closest sense embedding to the context vector, which is computed by averaging its contained words' embeddings. Sentence-level baselines. We also report results for two baseline models which view the task as context (sentence) similarity. The BoW system views the sentence as a bag of words and computes a simple embedding as average of its words. The system makes use of Word2vec (Mikolov et al., 2013b) 300d embeddings pre-trained 9 https://github.com/google-research/bert/blob/master/ 10 Multi-prototype embeddings are also referred to as sense embeddings in the literature. on the Google News corpus. Sentence LSTM is another baseline, which differently from the other models, does not obtain explicit encoded representations of the target word or sentence. The system has two LSTM layers with 50 units, one for each context side, which concatenates the outputs and passes that to a feedforward layer with 64 neurons, followed by a dropout layer at rate 0.5, and a final one-neuron output layer of sigmoid activation. We used two simple binary classifiers in our experiments on top of all comparison systems (ex-cept for the LSTM baseline). MLP : a simple dense network with 100 hidden neurons (ReLU activation), and one output neuron (sigmoid acti-vation), tuned on the development set (batch size: 32; optimizer: Adam; loss: binary crossentropy). Given the stochasticity of the network optimizer, we report average results for five runs (  standard deviation). Threshold : a simple threshold-based classifier based on the cosine distance of the two input vectors, tuned with step size 0.02 on the development set. Table 3 shows the results on WiC. In general, the dataset proves to be very difficult for all the techniques, with the best model, i.e., BERT large , providing around 14% absolute improvement over a random baseline. Among the two classifiers, the simple threshold-based strategy, which computes the cosine distance between the two encodings, proves to be more efficient than the MLP network which might not be suitable for this setting with small amount of training data. The 16 . 2% absolute accuracy difference between human-level upperbound and state-of-the-art performance suggests, however, a challenging dataset and encourages future research in context-sensitive word embeddings to leverage WiC in their evaluations. Among the contextualized word-based models, after BERT, Context2vec provides more competitive results on the dataset. However, surprisingly, neither ELMo nor Context2vec are able to improve over the simple sentence BoW baseline (which also outperforms the sentence LSTM baseline) using the threshold strategy. This raises a question about the ability of these models in capturing fine-grained semantics of words in various contexts. Finally, as far as multi-prototype techniques are concerned, DeConf is the best performer. We note that DeConf indirectly benefits from sense-level information from WordNet encoded in its embeddings. The same applies to SW2V, which leverages knowledge from a significantly larger lexical resource, i.e., BabelNet (Nav-igli and Ponzetto, 2012). The Stanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) comprises 2003 word pairs and is analogous to standard word similarity datasets, such as RG-65 (Rubenstein and Goode-nough, 1965) and SimLex (Hill et al., 2015), in which the task is to automatically estimate the semantic similarity of word pairs. Ideally, the estimated similarity scores should have high correlation with those given by human annotators. However, there is a fundamental difference between SCWS and other word similarity datasets: each word in SCWS is associated with a context which triggers a specific meaning of the word. The unique property of the dataset makes it a suitable benchmark for multi-prototype and contextualized word embeddings. However, in the following, we highlight some of the limitations of the dataset which hinder its suitability for evaluating existing techniques. Inter-rater agreement (IRA) is widely accepted as a metric to assess the annotation quality of a dataset. The metric reflects the homogeneity of ratings which is expected to be high for a well-defined task and a qualified set of annotators. For each word pair in SCWS ten scores were obtained through crowdsourcing. We computed the pairwise IRA to be 0.35 (in terms of Spearman  correlation) which is a very low figure. The mean IRA (between each annotator and the average of others), which can be taken as a human-level performance upperbound, is 0.52. Moreover, most of the instances in SCWS have context pairs with different target words. 14 This makes it possible to test context-independent models, which only considers word pairs in isolation, on the dataset. Importantly, such a context-independent model can easily surpass the human-level performance upperbound. For instance, we computed the performance of the Google News Word2vec pre-trained word embeddings (Mikolov et al., 2013b) on the dataset to be 0.65 (  ), which is significantly higher than the optimistic IRA for the dataset. In fact, Dubossarsky et al. (2018) showed how the reported high performance of multi-prototype techniques in this dataset was not due to an accurate sense representation, but rather to a subsampling effect, which had not been controlled for in similarity datasets. In contrast, a context-insensitive word embedding model would perform no better than a random baseline on our dataset. In this paper we have presented a benchmark for evaluating context-sensitive word representations. The proposed dataset, WiC, is based on lexicographic examples, which constitute a reliable basis to validate different models in their ability to perceive and discern different meanings of words. We tested some of the recent state-of-the-art contextualized and multi-prototype embedding models on our dataset. The considerable gap between the performance of these models and the human-level upperbound suggests ample room for future work on modeling the semantics of words in context. We would like to thank Luis Espinosa-Anke and Carla Perez-Almendros for their help with the manual evaluation and Kiamehr Rezaee for running the BERT experiments. 14 Only 8% (12% if ignoring PoS) of SCWS pairs are identical but their assigned scores (by average 6.8) are substantially higher than the dataset average of 3.6 on a [0,10] scale.
abstain abstain abstain objective abstain other abstain abstain other abstain abstain abstain abstain method abstain objective other abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other abstain other other other method other other other other method other objective method abstain method abstain other other