LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond

Distributional semantics based on neural approaches is a cornerstone of Natural Language Processing, with surprising connections to human meaning representation as well. Recent Transformer-based Language Models have proven capable of producing contextual word representations that reliably convey sense-specific information, simply as a product of self-supervision. Prior work has shown that these contextual representations can be used to accurately represent large sense inventories as sense embeddings, to the extent that a distance-based solution to Word Sense Disambiguation (WSD) tasks outperforms models trained specifically for the task. Still, there remains much to understand on how to use these Neural Language Models (NLMs) to produce sense embeddings that can better harness each NLM's meaning representation abilities. In this work we introduce a more principled approach to leverage information from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We also emphasize the versatility of these sense embeddings in contrast to task-specific models, applying them on several sense-related tasks, besides WSD, while demonstrating improved performance using our proposed approach over prior work focused on sense embeddings. Finally, we discuss unexpected findings regarding layer and model performance variations, and potential applications for downstream tasks.

Introduction
Lexical ambiguity is prevalent across different languages and plays an important role in improving communication efficiency (Piantadosi et al., 2012). Word Sense Disambiguation (WSD) is a long-standing challenge in the field of Natural Language Processing (NLP), and Artificial Intelligence more generally, with an extended history of research in computational linguistics (Navigli, 2009).
Interestingly, both computational and psychological accounts of meaning representation have converged on high-dimensional vectors within semantic spaces.
From the computational perspective, there is a rich line of work on learning word embeddings based on statistical regularities from unlabeled corpora, following the well-established Distributional Hypothesis (Harris, 1954;Firth, 1957, DH). The first type of distributional word representations relied on count-based methods, initially popularized by LSA (Deerwester et al., 1990), and later refined with GloVe (Pennington et al., 2014). Before GloVe, word embeddings learned with neural networks, first introduced by Bengio et al. (2003a), gained wide adoption with word2vec (Mikolov et al., 2013a) and, afterwards, culminated with fastText (Bojanowski et al., 2017). The development and improvement of word embeddings has been a major contributor to the progress of NLP in the last decade (Goldberg, 2017).
From the psychological perspective, there is also ample behavioural evidence in support of distributional representations of word meaning. Similarly to word embeddings, these representations are related according to the degree of shared features within semantic spaces, which translates into proximity in vector-space (Rodd, 2020;Klein & Murphy, 2001). Understandably, the nature of the features making up this psychological account of semantic space, among other aspects (e.g., learning method), is not as clear as we find in the computational account.
Nevertheless, contextual co-occurrence is among the most informative factors for meaning representation as well (Mcdonald & Ramscar, 2001;Erk, 2016;Radach et al., 2017). There are even use cases in neurobiology motivating research into accurate distributional representations of word meaning. In Pereira et al. (2018), word embeddings have proven useful for decoding words and sentences from brain activity, after learning a mapping between corpus-based embeddings (i.e., GloVe and word2vec) and fMRI activation.
The current understanding of how humans perform disambiguation attributes major relevance to sentential context, and other linguistic and paralinguistic cue's (e.g., speaker accent) to a lesser extent (Rodd, 2020;Cai et al., 2017). However, the previously mentioned computational approaches are not designed for sense-level representation due to the Meaning Conflation Deficiency (Camacho-Collados & Pilehvar, 2018), as they converge different senses into the same word-level representation. Some works have explored variations on the word2vec method for sense-level embeddings (Rothe & Schütze, 2015;Iacobacci et al., 2015;Pilehvar & Collier, 2016;Mancini et al., 2017), but the dynamic word-level interactions composing sentential context were not targeted by those works.
The works of Melamud et al. (2016); Yuan et al. (2016); Peters et al. (2018a) were among the first to propose Neural Language Models (NLMs) featuring dynamic word embeddings conditioned on sentential context (i.e., contextual embeddings). These works showed that NLMs (trained exclusively on language modelling objectives) can produce contextual embeddings for word forms that are sensitive to the word's usage in particular sentences. Furthermore, these works also addressed WSD tasks with a simple nearest neighbours solution (k-NN) based on proximity between contextual embeddings. Their results rivalled systems trained specifically for WSD (i.e., with additional modelling objectives), highlighting the accuracy of these contextual embeddings.
However, it was not until the development of Transformer-based NLMs, namely BERT (Devlin et al., 2019), that contextual embeddings from NLMs showed clearly better performance on WSD tasks than previous systems trained specifically for WSD (LMMS, Loureiro & Jorge, 2019a).
In this earlier work, we explored how to further take advantage of the representational power of NLMs through propagation strategies and encoding sense definitions. Besides pushing the state-of-the-art of WSD, in Loureiro & Jorge (2019a) we created sense embeddings for every entry in the Princeton Word-Net v3.0 (200k word senses, Fellbaum, 1998), so that the semantic space being represented is granular and expansive enough to encompass general knowledge domains for various parts-of-speech of the English language. With this fully populated semantic space at our disposal we suggested strategies for uncovering biases and world knowledge represented by NLMs.
Since our work on LMMS, others have shown additional performance gains for WSD with fine-tuning or classification approaches that make better usage of sense definitions (Huang et al., 2019;Blevins & Zettlemoyer, 2020), semantic relations from external resources (Scarlini et al., 2020a;Bevilacqua & Navigli, 2020), or altogether different approaches to WSD (Barba et al., 2021).
However, there are several questions still standing regarding how to leverage NLMs for creating accurate and versatile sense embeddings, beyond optimizing for WSD benchmarks only. Given that semantic spaces with distributional representations of word meanings feature prominently in both the conventional computational and psychological accounts of word disambiguation, these questions warrant further exploration.
Contributions. In this extension of LMMS, we broaden our scope to more recent Transformer-based models in addition to BERT Liu et al., 2019b;Lan et al., 2020) (14 model variants in total), verify whether they exhibit similar proficiency at sense representation, and explore how performance variation can be attributed to particular differences in these models. Striving for a principled approach to sense representation with NLMs, we also introduce a new layer pooling method, inspired by recent findings of layer specialization (Reif et al., 2019), which we show is crucial to effectively use these new NLMs for sense representation. Most importantly, in this article we provide a general framework for learning sense embeddings with Transformers and perform an extensive evaluation of such sense embeddings from different NLMs on various sense-related tasks, emphasizing the versatility of these representations.
The method used to produce this work's sense embeddings is described in Section 4, covering aspects of the method introduced in Loureiro & Jorge (2019a) (from §4.1 to §4.3), as well as our new layer pooling method in Section 4.4.
In Section 5 we describe our experimental setting, providing relevant details about our choice of NLMs ( §5.1) and annotated corpora used to learn sense embeddings ( §5.2).
The layer pooling methodology described in Section 4.4 requires validating performance under two distinct modes of application. Consequently, in Section 6 we report on performance variation per layer across NLMs ( §6.1), highlight differences between disambiguation and matching profiles ( §6.2), and present the rationale for choosing particular profiles for each task ( §6.3).
In order to better understand the contributions of this work, Section 8 reports on several ablation analyses targeting the following: choice of Sense Profiles ( §8.1), impact of unambiguous word annotations ( §8.2), merging gloss representations ( §8. 3), and indirect representation of synsets ( §8.4).
We discuss our findings in Section 9, regarding representations from intermediate layers of NLMs ( §9.1), irregularities across models and variants ( §9.2), and potential downstream applications of our sense embeddings focusing on knowledge integration ( §9.3).
Finally, in Section 10 we present our concluding remarks, and provide details about our release of sense embeddings, code and more.

Preliminaries
This work exploits the interaction between vector-based semantic representations ( §2.1), recent developments on NLMs ( §2.2), and curated sense inventories ( §2.3). In this section we provide some background on these topics.

Vector Semantics
Nearly a century ago, Firth (1935) postulated that "the meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously". Indeed, after working on formal theories of word meaning definition, Wittgenstein (1953) conceded "the meaning of a word is its use in a language". This view of meaning representation became known as the Distributional Hypothesis (Harris, 1954, DH), which proposes that words that occur in the same contexts tend to have similar meanings. During this period, Osgood et al. (1957) further proposed representing the meaning of words as points in multidimensional space, with similar words having similar representations, thus being placed closely in this space. Still, it would take a few more decades of computing advancements to appreciate the implications of the DH.
Early VSMs. After some early works introducing vector space models (VSMs) for information retrieval (Salton 1971;1975), Deerwester (19891990) was the first to use dense vectors to represent word meaning, initially with a method called Latent Semantic Indexing (LSI), and later with Latent Semantic Analysis (LSA). LSA was based on a word-document weighted frequency matrix from which the first 300-dimensions resulting from Singular-Value Decomposition (SVD) would correspond to word embeddings. Lund & Burgess (1996) introduced another influential method similar to LSA, called Hyperspace Analogue to Language (HAL) which differed from LSA by considering word-word frequencies instead, introducing the notion of a fixed-sized window as context (e.g., the two words to the left and to the right) instead full documents, which would become the standard representation of context. Following these developments, Landauer & Dumais (1997) evaluated the performance of LSA embed-dings learned from large corpora on a simple semantic task (synonymy tests) and found that these embeddings performed comparably to school-aged children, when measuring similarity between word pairs as the cosine similarity between their corresponding embeddings (inspired by applications for information retrieval). Already in this early period, Schutze (1992) and Yarowsky (1995) realized the potential for WSD applications based on the similarity between unsupervised word embeddings. Blei et al. (2003) would later introduce Latent Dirichlet Allocation (LDA) which uses a generative probabilistic approach to generalize and improve on the approach used for LSA, being widely adopted for topic modelling and other applications beyond semantic analysis.
Neural Models. Having established that corpus-based word embeddings are able to capture semantic knowledge, additional progress followed swiftly. A milestone in the evolution of word embeddings was the discovery that Neural Language Models (NLMs) implicitly develop word embeddings when training for the task of word prediction (Bengio et al., 2003b). Shortly after, Collobert 2007Collobert , 2008Collobert , 2011 demonstrated that word embeddings could be incorporated into neural architectures for various NLP tasks. With word2vec, Mikolov et al. (2013b) distilled the components of NLMs responsible for learning word embeddings into a lightweight and scalable solution, allowing this neural-based solution to be employed on corpora of unprecedented size (100B tokens). Nevertheless, countbased solutions would still remain important, particularly GloVe (Pennington et al., 2014), as these methods were also significantly improved. The next major improvement was the introduction of fastText (Bojanowski et al., 2017), which was able to represent words absent from training data by leveraging subword information, as well as refining several aspects of word2vec's training method.
Sense Embeddings. In spite of their success, word2vec, GloVe and fastText conflated different senses of the same word form into the same representation, a shortcoming known as the Meaning Conflation Deficiency (Camacho-Collados & Pilehvar, 2018). While a number of extensions were proposed for the creation of sense-specific representations, such as AutoExtend (Rothe & Schütze, 2015), NASARI (Camacho-Collados et al., 2016), DeConf (Pilehvar & Collier, 2016) or Probabilistic FastText (Athiwaratkun et al., 2018), this issue would require the development of a new generation of NLMs in order to be effectively addressed.

Neural Language Modelling
The first major step towards contextual embeddings from NLMs, was the development of context2vec (Melamud et al., 2016), a single-layer bidirectional LSTM trained with the objective of maximizing similarity between hidden states and target word embeddings, similarly to word2vec. Peters et al. (2018a) built upon context2vec with ELMo, a deeper bidirectional LSTM trained with language modelling objectives that produce more transferrable representations.
Both context2vec and ELMo emphasized WSD applications, providing the most convincing accounts until then that sense embeddings can be effectively represented as centroids of contextual embeddings, showing 1-NN solutions to WSD tasks that rivalled the performance of task-specific models.
With the introduction of highly-scalable Transformer architectures (Vaswani et al., 2017), two kinds of very deep NLMs emerged: causal (or left-to-right) models, epitomized by the Generative Pre-trained Transformer (Brown et al., 2020, GPT-3), where the objective is to predict the next word given a past sequence of words; and masked models, where the objective is to predict a masked (i.e., hidden) word given its surrounding words, of which the most prominent example is the Bidirectional Encoder Representations from Transformers (Devlin et al., 2019, BERT). The difference in training objectives results in these two varieties of NLMs specializing at different tasks, with causal models excelling at language generation and masked models at language understanding. 1 BERT proved highly successfully at most NLP tasks (Rogers et al., 2020), and motivated the development of numerous derivative models, many of which we also explore in this work. In spite of this progress, Transformer-based NLMs can still show strong reliance on surface features (McCoy et al., 2019) and social 1 Although recent models like BART (Lewis et al., 2020) show progress towards both. biases which are hard to correct (Zhou et al., 2021). There are known theoretical limits to how much language understanding can be expected from models trained with language modelling objectives alone (Bender & Koller, 2020;Merrill et al., 2021), and it is not clear how far current models are from those limits.

Sense Inventories
The currently most popular English word sense inventory is the Princeton WordNet (Fellbaum, 1998) (henceforth, WordNet), a large semantic network comprised of general domain concepts curated by experts 2 .
The core unit of WordNet is the synset, which represents a cognitive concept.
Each lemma (word or multi-word expression) in WordNet belongs to one or more synsets, and word senses amount to the combination of word forms and synsets (referred as sensekeys). As a result, the set of words that belong to a synset can be described as synonyms, with some words being ambiguous (belonging to additional synsets) while others not (specific to a synset). The predominant semantic relation in WordNet, which relates synset pairs, is hypernymy (i.e., Is-A). Each synset also features a gloss (dictionary definition), part-of-speech (noun, verb, adjective or adverb) and lexname 3 , which is a syntactic category and logical grouping. Synsets are formally represented as numerical codes. Following related works, we also represent them using the more readable format lemma # P OS , where lemma corresponds to synset's most representative lemma. As an example, the lemma 'mouse' is polysemous belonging to the mouse 1 n (rodent) and mouse 4 n (computer mouse) synsets, among others. The most frequent sense for mouse, mouse%1:05:00:: (sensekey), belongs to the synset mouse 1 n (02330245n) which has an hypernymy relation with rodent 1 n , lexname 'noun.animal', and gloss "any of numerous small rodents typically [...]".

Related Work
In this section we cover related work on the various well-researched topics that our work intersects, namely Sense Embeddings ( §3.1), WSD ( §3.2) and Probing NLMs ( §3.3).

Sense Embeddings
Sense embeddings emerged in NLP due to the so-called meaning conflation deficiency of word embeddings (Camacho-Collados & Pilehvar, 2018). By merging several meanings into a single representation, the single vector proved insufficient in certain settings (Yaghoobzadeh & Schütze, 2016), and contradicted common laws in distance metrics, such as the triangle inequality (Neelakantan et al., 2014). In order to solve this issue, the field of sense vector representation mainly split into two categories: (1) unsupervised, where senses were learned directly from text corpora (Reisinger & Mooney, 2010;Huang et al., 2012;Vu & Parker, 2016); (2) or knowledge-based, where senses were linked to a predefined sense inventory by exploiting an underlying knowledge resource (Rothe & Schütze, 2015;Pilehvar & Collier, 2016;Mancini et al., 2017;Colla et al., 2020a).
In this article, we focus on the latter type of representation, particularly leveraging powerful Transformer-based language models trained on unlabeled text corpora. As such, the final representation is mainly constructed based on the knowledge learned by the language models, and knowledge resources such as WordNet serve to guide the annotation process. The goal of this paper is indeed to construct a task-agnostic sense representation that can be leveraged in semantic and textual applications. This differs from traditional static sense embeddings which, with a few notable exceptions (Li & Jurafsky, 2015;Flekova & Gurevych, 2016;Pilehvar et al., 2017), were mainly leveraged in intrinsic sensebased tasks only. As we show throughout this paper, general-purpose sense representations learned with the power of Transformers and guided through an underlying lexical resource such as WordNet prove to be robust in a range of text-based semantic tasks, as well as in intrinsic sense-based benchmarks.

Word Sense Disambiguation
As one of the earliest Artificial Intelligence tasks, WSD has a long history of research. In this work, our coverage of related work for WSD is focused on recent systems using Transformer-based architectures for two reasons: our own experiments are also focused on Transformer-based systems; the current stateof-the-art for WSD has converged on these systems. Additionally, we also distinguish between solutions addressing WSD from the nearest neighbors paradigm, using pre-computed sense embeddings, and task-specific solutions fine-tuning Transformer models or training classifiers using their internal representations.

Nearest Neighbors
Our prior LMMS work (described throughout this paper) was the first to demonstrate that a nearest neighbors solution based on sense embeddings pooled from internal representations of BERT (i.e., feature extraction) could clearly outperform the state-of-the-art of the time, which still had not adopted Transformerbased models.
SensEmBERT (Scarlini et al., 2020a) followed a similar approach to LMMS, but leveraged BabelNet to reduce dependency on annotated corpora, producing sense embeddings that performed better on WSD, though limited to nouns only.
With ARES, Scarlini et al. (2020b) introduce a method to produce a large number of semi-supervised annotations to dramatically increase the coverage of the sense inventory, and demonstrated that sense embeddings learned from those annotations can perform substantially better on WSD than LMMS.
SensEmBERT and ARES use the same layer pooling method and gloss embeddings as LMMS, although both have employed not only BERT-L, but also its multilingual variant, showing strong performance on languages other than English as well.
In addition to WSD, to our knowledge, the only other task these works have applied their sense embeddings is Word-in-Context (Pilehvar & Camacho-Collados, 2019, WiC), which we also address in this work.

Trained Classifiers
When it comes to using Transformers to train classifiers specific to the WSD task, we encounter a much more diverse set of solutions in comparison to feature extraction approaches.
One of the earliest and most straightforward supervised classifiers for WSD using BERT was the Sense Vocabulary Compression (SVC) of Vial et al. (2019), which added layers to BERT, topped with a softmax classifier, to be trained targeting a strategically reduced set of admissible candidate senses.
Following outstanding results on a range of text classification tasks by model fine-tuning, GlossBERT (Huang et al., 2019) fine-tuned BERT using glosses so that WSD could be framed as a text classification task pairing glosses to words in context. KnowBERT (Peters et al., 2019) employs a more sophisticated finetuning approach, designed to exploit knowledge bases (WordNet and Wikipedia) as well as glosses.
Straying further from prototypical classifiers, Blevins & Zettlemoyer (2020) (BEM) propose a bi-encoder method which learns to represent senses based on glosses while performing the optimization jointly with the underlying BERT model. Taking advantage of an ensemble of sense embeddings from LMMS and SensEmBERT, along with additional resources, EWISER (Bevilacqua & Navigli, 2020) trains a multifaceted high performance WSD classifier.
Finally, the current state-of-the-art for WSD is ConSeC (Barba et al., 2021), which obtains impressively strong results by framing WSD as an extractive task, similar to extractive question answering, trained through fine-tuning BART (Lewis et al., 2020), a sequence-to-sequence Transformer which outperforms BERT on reading comprehension tasks (while being of comparable size).
In Loureiro et al. (2021) we extensively compared fine-tuning and feature extraction approaches for the WSD task. Consistent with prior work, we found that fine-tuning overall outperforms feature extraction. However, under comparable circumstances, the performance gap is narrow and feature extraction shows improved few-shot performance and less frequency bias.

Probing Neural Language Models
As NLMs became popular, investigating properties of their internal states, or intermediate representations, also became an important line of research, often referred to as 'model probing'. Probing operates under the assumption that if a relatively simple classifier, based exclusively on representations from NLMs, can perform well at some task, then the required information was already encoded in the representations. For clarity, we define probes as functions (learned or heuristic) designed to reveal some intrinsic property of NLMs. In this section we cover probing works focused on lexical semantics and layer-specific variation that inspired our probing analysis. We distinguish these works by their use of probes trained using representations (learned), and probes directly comparing or analysing unaltered representations (heuristics, such as nearest neighbors).

Learned Probes
Among the most influential findings in this line of research was the discov-  et al. (2018b) concluded that top layers can be less effective for semantic tasks possibly due to specialization for the language modelling tasks optimized during pre-training. Tenney et al. (2019b) proposed an "edge probing" methodology, using scalar mixing, that allowed for evaluating different syntactic or semantic properties using a common classifier architecture, where probing models are trained to predict graph edges independently. In Tenney et al. (2019a), edge probing was employed to reveal that BERT implicitly performed different steps of a traditional NLP pipeline, in the expected order as information flows through the model, with lower layers processing local syntax (e.g., Part-of-Speech) and higher layers processing complex semantics of arbitrary distance (e.g., Semantic Roles). Raising concerns about remaining faithful to the information encoded in the representations, Kuznetsov & Gurevych (2020) proposes reducing the expressive power of learned probes while improving edge probing. Liu et al. (2019a) ran several probing experiments with simpler probes (i.e., linear classifiers), investigating differences between NLM architectures, namely ELMo, GPT and BERT, while still finding competitive performance with state-of-the-art task-specific models. They confirm that LSTM-based models (i.e., ELMo) present more task-specific (less transferable) top layers, but Transformers-based models (i.e., BERT) are less predictable and do not exhibit monotonic increase in task-specificity, in line with our own findings. GPT was found to significantly underperform ELMo and BERT, which Liu et al. (2019a) attributes to the fact that GPT is trained unidirectionally (left-to-right), while ELMo and BERT are trained bidirectionally.

Representational Similarity
Without recourse to learned probes, Ethayarajh (2019) investigated differences between ELMo, GPT-2 and BERT, relying on experiments based on cosine similarity to learn about the context-specificity of their representations. Etha-yarajh (2019) found that top layers show highest degree of context-specificity, but all layers of all three models produced highly anisotropic representations, with directions in vector space confined to a narrow cone, concluding that this property is an inherent consequence of the contextualization process. The anisotropy observed for all contextualized NLMs also supports the hypothesis of Reif et al. (2019) that sense-level information is encoded in a low-dimensional subspace, since contextualization is crucial for sense disambiguation. Vulić et al. (2020) reached similar conclusions regarding the detrimental contribution of top layers for lexical tasks (e.g., lexical semantic similarity) while also finding improved results from averaging different layers, particularly task-specific layer subsets, prompting further research into layer weighting or meta-embedding approaches, and motivating the present work. Through direct comparison of cosine similarities, Chronis & Erk (2020) reached similar conclusions as Vulić et al. (2020) about the role of top layers for lexical similarity tasks, adding that top layers appear to better approximate relatedness than similarity.

Method
We propose a principled approach for sense representation based on contextual NLMs trained exclusively with self-supervision. This approach is an extension of Loureiro & Jorge (2019a), addressing relevant issues still largely unresolved, particularly the influence of embeddings from the different layers composing NLMs, with the introduction of a novel layer probing methodology.
Moreover, in this work, we reinforce the distinction between sense disambiguation and sense matching by introducing methodological differences specific to each application scenario.

Learning Sense Embeddings
The initial process to learn sense embeddings is based on sense-annotated sentences and contextualized embeddings of annotated words or phrases in con- text. An overview of the process can be seen at Figure 1.
Formally, in order to generate sense embeddings learned in context from natural language, we require a pre-trained contextual NLM Ω (frozen parameters) and a corpus of sense-annotated sentences S. Every sense ψ is represented from the set of contextual embeddings c l ∈ C ψ , obtained by employing Ω on the set of sentences S ψ annotated with that sense (considering only contextual embeddings specific to tokens annotated with sense ψ), using representations at each layer l ∈ L, such that: The L set of layers typically used for sense representation is the last four [−1, −2, −3, −4] (reversed layer indices), as discussed in Section 3.1.

S
[CLS] The mouse is unplugged. [SEP] [CLS] I prefer trackpads to mice. [SEP] [CLS] This mouse has no batteries. [SEP] Layer Pooling C Ω Contextual NLM (pretrained w/frozen parameters) Figure 1: Overview of learning sense embeddings from annotated corpora. Showing how the sense ψ for 'computer mouse' is determined from a set for sentences annotated with that sense S ψ (padded with special tokens as expected by the NLM Ω). After pooling contextual embeddings C ψ from layers L, the sense embedding for ψ is computed as the centroid of C ψ .
Contextual NLMs typically operate at the subword-level, so the token-level embeddings c produced by Ω correspond to the average of each token's subword contextual embeddings (depending on the NLM, these may be BPE or WordPiece embeddings). Similarly, whenever sense-annotations cover a span of several tokens, we also use the average of the corresponding token-level embeddings as the contextual embedding. Contextual NLMs are pre-trained using special tokens at specific locations, so we also include these tokens in their expected positions (e.g., [CLS] at the start and [SEP] at the end with BERT).
As described in Section 2.3, WordNet can be used to represent senses in two ways: sensekeys and synsets. Sense-annotated corpora most often use sensekey annotations, so in those cases sensekey embeddings do not require any intermediate mapping. Synset embeddings can be derived from sensekey annotations in at least two ways, which we differentiate as 'direct' and 'indirect'. In the direct approach, each sensekey annotation is converted (mapped) to the corresponding synset, so synset representations are learned from each annotation instance. In the indirect approach we first learn sensekey-level embeddings, without converting annotations, and afterwards compute synset embeddings as the average of corresponding sensekey embeddings. The latter approach has been explored in earlier works in sense embeddings (Rothe & Schütze, 2015). In this work we explore both approaches.

Extending Coverage with Additional Resources
Given that one of the major issues in supervised WSD is the lack of sense annotations (Pasini, 2020), not just in their quantity but also in terms of their coverage of the sense inventory, we require solutions to address this in our method.
On this matter, we also follow the methods we first proposed in Loureiro & Jorge (2019a) and later optimized with the introduction of the UWA corpus in Loureiro & Camacho-Collados (2020). The two methods, ontological propagation and gloss representation, are designed to reach full coverage of the sense inventory, and they are complementary by exploiting different resources, namely semantic relations between senses and glosses (combined with lemmas).

Ontological Propagation
In Section 2.3 we introduced WordNet and the different elements and relations composing this semantic network. The ontological propagation method we presented in Loureiro & Jorge (2019a) exploits these relations between senses in WordNet in order to infer embeddings for senses which may not occur in annotated corpora. It is possible to infer accurate sense embeddings from these relations due to the fine-granularity of WordNet, along with widespread synonymy and hypernymy relations, to the extent that in Loureiro & Camacho-Collados (2020) we showed that even annotations for unambiguous words can significantly improve the propagation process.
Since available corpora do not provide full-coverage annotations for our sense inventory of interest, by following the process described in Section 4.1 we are left with a represented senses Ψ, and a set of unrepresented senses Ψ . The propagation process involves three steps, using increasingly abstract relations from WordNet -sets of synonyms (synsets), hypernymy relations, and lexical categories (lexnames or supersenses). In case we are targeting synset-level representations, then the first step/level should be skipped.
Considering we are provided mappings between sensekeys, synsets, hypernyms and lexnames, we infer Ψ iteratively following Algorithm 1. After each of these sequential steps, every inferred ψ is added to the set of represented senses Ψ. This propagation method ensures full-coverage provided that initial sense embeddings Ψ are sufficiently diverse such that falling back on propagating from lexnames (supersenses) is always possible.
Since this method is designed to achieve full-representation of the sense inventory based on a subset of senses observed in context, the inferred representations are also of a similar contextual nature. However, unless the initial set of sense embeddings is nearly complete, and particularly diversified, this propagation method produces some number of identical representations for distinct senses, which is most undesirable for disambiguation applications, and to a lesser extent, sense matching applications as well.
Algorithm 1: Propagation method to infer unrepresented senses Ψ , using sense embeddings Ψ learned from annotations, and relations R.

Leveraging Glosses and Lemmas
In Loureiro & Jorge (2019a) we introduced a method for representing sense embeddings based on glosses and lemmas. This method is inspired by a typical baseline approach used in works pertaining to sentence embeddings, and it amounts to simply averaging the contextual embeddings for all tokens present in a sentence. In our case, we use glosses as sentences, but also introduce lemmas into the gloss' context. By combining glosses with lemmas, we not only augment the information available to represent senses, but we are also able to generate sense embeddings which are lemma-specific (sensekey-level), instead of only concept-specific (synset-level) if we only used glosses. As such, sense embeddings generated by this method address the redundancy issue arising from the previously described propagation method, while simultaneously introducing representational information which is complementary to contextual embeddings extracted from sense-annotated sentences.
The method proceeds as follows. For every lemma/sense pair (i.e., sensekey) in a sense inventory, we build the template "<lemma> , <sense lemmas> -<sense gloss>". For instance, based on WordNet, the synset race v 2 has the lemmas race and run which are provided with the following sensekey-specific fill-outs of the template: • race%2:33:00:: -"race -run, race -compete in a race" • run%2:33:01:: -"run -run, race -compete in a race" The initial "<lemma>" component of the template can be omitted if the target representation level is synsets, as it only serves the purpose of reinforcing the lemma which is specific to the sensekey. The templated string is processed by Ω, similarly to sentences S in Section 4.1, but here we use the resulting set of contextual embeddings for every token C .
Considering that we have a complete set of sense embedding Ψ, based on sense annotations and propagation as described in Sections 4.1 and 4.2.1, we augment ∀ ψ ∈ Ψ with gloss and lemma information as follows: In contrast to Loureiro & Jorge (2019a), which proposed using concatenation to merge this new set of sense embeddings based on glosses and lemmas with the previously mentioned set, in this work we propose merging through averaging instead. This departure is motivated by the fact that Loureiro & Jorge (2019a) found that while concatenation outperformed averaging for WSD, the difference in performance was modest, and in this work we are interested in additional tasks which that work did not cover. Merging representations through concatenation doubles the dimensionality of sense embeddings, increasing computational requirements and complicating comparison with contextual embeddings, among other potential applications. On the other hand, merging representations through averaging allows for adding more components while retaining a similar vector, of equal dimensionality to contextual embeddings, and represented in the same vector space.

Applying Sense Embeddings
In this section we address how sense embeddings can be employed for solving various tasks, grouped under two paradigms: disambiguation and matching.
Disambiguation assigns a word in context (i.e., in a sentence) to a particular sense out of a subset of candidate senses, restricted by the word's lemma and part-of-speech. Matching also assigns specific senses to words, but imposes no restrictions, admitting every entry in the sense inventory for each assignment.
The different conditions for disambiguation and matching require sense representations with different degrees of lexical information and semantic coherence. Whereas, for disambiguation, lexical information can be absent from sense representations, due to the subset restrictions, for matching, lexical information is essential to distinguish between word forms carrying identical or similar semantics. Similarly, the disambiguation setting has no issues with sense representations displaying inconsistencies such as eat being more similar to sleep than to drink, since these all belong to disjoint subsets, but the order and coherence of these similarities is relevant for sense matching applications. This distinction leads us to specialize sense embeddings accordingly in Section 4.4.
To disambiguate a word w in context, we start by creating a set Ψ w of candidate senses based on its lemma and part-of-speech, using information provided with the sense inventory. Afterwards, we compute the cosine similarities (denoted 'cos') between the word's contextual embedding c w and the pre-computed embeddings for each sense in this subset Ψ w (both using the same layer pooling).
Finally, we assign the sense whose similarity is highest (i.e., nearest neighbor): To match a word w in context, without restrictions, we follow the approach for disambiguation but simply consider the full sense inventory Ψ instead of Ψ w :

Grounding Layer Pooling
Up until this point, we have described our method closely following our prior work in Loureiro & Jorge (2019a). As we covered in earlier sections, NLMs can show substantial and, more importantly, unexpected variation in task performance across their layers. Considering this work's focus on a more principled and grounded focus on sense representation with NLMs, our methodology also covers this important aspect.
In this section we present two methods targeting the layers composing NLMs.
The first method probes each layer's adeptness for sense representation. Consequently, the second method in this section is designed to capitalize on that knowledge towards sense representations which better capture NLM's ability to represent senses over the current paradigm.
As we alluded to in Section 4.3, sense representation should be viewed in light of the intended applications for these representations. In particular, in this work we differentiate between representations used for disambiguating words, and for matching or comparing senses. This distinction is motivated by the fact that disambiguation, which is the prevalent sense-related task on NLP, only requires that sense representations be adequately differentiated between the restricted set of senses which share the same lemmas and parts-of-speech.
However, there exist other potential applications where sense representations are matched without any constraints on the sense inventory, and thus require that senses be coherently represented across the semantic space.

Sense Probing
In order to assess the contribution of individual layers of a pre-trained NLM for sense representation, we directly evaluate the performance of representations from these layers on tasks related to the previously described disambiguation or matching scenarios. These tasks are solved using the nearest neighbors approaches described in Section 4.3, comparing pre-computed sense embeddings with contextual embeddings obtained from the same layer.
For this probing experiment, we follow the method for learning sense representations described in Section 4.1, but create multiple sets of senses Ψ l for each layer l in the NLM. To maintain focus on assessing the performance of representations learned directly from specific layers, we ensure that test instances have all their senses represented in the sense-annotated corpora used to precompute Ψ l , ∀l ∈ L. Thus, our probing experiments do not use techniques to infer or enrich sense representations, such as those we described in Section 4.3, which could otherwise act as confounders.
The resulting performance scores for every layer l ∈ L composing a specific Ω, using a corresponding Ψ l , not only reveal which layers perform best, but also inform the layer pooling method described next.

Sense Profiling
We use the probing results described earlier as the basis for a pooling operation which is better grounded than the current paradigm of using the sum of the last four layers, and also better performing as we show later in this work.
We designate each set of model-specific layer weights as a 'sense profile', and consider distinct sense profiles for disambiguation and matching, depending on the choice of disambiguation or matching tasks during layer probing.
These proposed sense profiles are a more immediate version of the Scalar Mixing used in Tenney et al. (2019a), being based on heuristically-derived sets of layer weights, instead of learning them through task optimization. Considering this, we understand sense profiles to be closer to the extraction configurations of Vulić et al. (2020).
Granted we have performance scores s l ∀l ∈ L, for a specific Ω, we obtain layer specific weights w l ∀l ∈ L by applying the softmax function: We use the temperature scaling parameter t to skew the weight distribution towards highest performing layers. While simple, temperature scaling has been found surprisingly effective at calibrating neural network predictions (Guo et al., 2017). This parameter is to be determined empirically and is only specific to application settings, not models.
In Table 1 we demonstrate the interaction between performance scores and layer weights conditioned on the temperature parameter. In that table, and others found in this work, we use reverse layer indices so that we can consistently refer to the final layer of any model using the -1 index, regardless of the number of layers in the NLM.
Consequently, we employ sense profiles comprised of weights w l ∀l ∈ L to retrieve contextual embeddings from Ω, and generate our sense embeddings accordingly, updating formula (1) such that: This set of sense embeddings learned from annotations using sense profiles, undergoes the same extensions and augmentations described earlier ( §4.2).
To be clear, the process of probing layer performance and determining sense

Experimental Setting
In this section we provide details about our experimental setting, including a description of the models ( §5.1) and datasets ( §5.2) used for learning sense representations.

Transformer-based Language Models
In this work we experiment with several Transformer-based Language Models, including all the original English BERT models released by Devlin et al.
(2019) as well as several other BERT-inspired alternatives, namely XLNet , RoBERTa (Liu et al., 2019b) and ALBERT (Lan et al., 2020). This section briefly describes the most relevant features of each of these models for our use case. We summarize the differences between each variant of these models on

Corpora for Training and Validation
We learn the initial set of sense representations described in Section 4.1 using sense-annotated corpora, namely SemCor (Miller et al., 1994)

Probing Analysis
In this section we present the outcome of the probing methodology described in Section 4.4 applied on the models detailed in Section 5.1. We report probing results in this section so they are presented and discussed before the evaluation and analysis sections, which report downstream task results using layer pooling informed by the probing analysis.
This section starts by covering our initial findings regarding layer performance variation patterns observed for all models ( §6.1). Second, we present validation results using our proposed sense profiles for both disambiguation and matching scenarios ( §6.2). Finally, we present our rationale for choosing which sense profile should be used according to the type of task ( §6.3).

Variation in Layer Performance Across NLMs
As discussed in Section 3.3, it is well-understood that task performance varies considerably depending on which layers are used to retrieve embeddings from.
While some works have analysed task performance per layer specifically for the task of WSD (Reif et al., 2019;Loureiro et al., 2021), there is still lacking an in-depth cross-model comparison.
In Table 3 we report WSD and USM performance for individual layers of each of the 14 models belonging to 4 different Transformer-based model families.
These results are obtained using the methodology described in Section 4.4.1.
We observe that indeed, the final layer (-1) is never optimal for either WSD or USM performance. More interestingly, we find that while the second-to-last layer (-2) always performs best for WSD in BERT models (with the exception of BERT-L-UNC-WHL), that pattern does not hold for the other models tested.
For some models, such as XLNet-L or ALBERT-L, we even find that the best performing layers are closer to the initialization layer (INIT) than to the final layer. Another apparent pattern is that the best performing layers for USM are consistently lower than for WSD. This can be explained by the fact that USM benefits from lexical information encoded in the lower layers, even though the initialization layer still performs worst, just as with WSD, demonstrating that it is not sufficient by itself.
These empirical results suggest that any layer pooling strategy based on a fixed set of layers, such as the often used sum of layers [-1,-4], cannot accurately capture the available sense information encoded in pre-trained NLMs.

Word Sense Disambiguation (WSD)
Model   Table 3: Performance variation on the development set across all models and configurations considered in this work. Green represents best performing layers (best is marked with ), red represents worst performing layers, and grey stands for layers missing in shallower variants.

Sense Profiles for Disambiguation and Matching
In Section 4.4.2 we described our method for uncovering a model-specific set of layer weights which informs a weighted layer pooling that results in improved sense representations. We have applied this method to all our models, for both disambiguation and matching scenarios, in order to verify whether our proposed method reliably improves performance on WSD and USM tasks when compared to conventional pooling approaches. Additionally, we also compare against different values for the temperature t parameter. In order to understand whether our recommended t values actually result in improved test-time performance for these tasks, we run a limited evaluation on the ALL test set of Raganato et al.
(2017) where we compare NLMs using only our method as described until Section 4.2.1 (without using glosses) and trained solely with SemCor annotations.
Later, in Sections 7.1 and 7.2, we report WSD and USM results using our final solution in comparison with the current state-of-the-art.
The WSD validation results on Table 4 reveal that the single best layer (which varies depending on model, see Table 3) consistently outperforms the sum of last 4 layers. We also find that WSD sense profiles with t = 0.002 and t = 0.005 perform comparably to the single best layer, with t = 0.002 being slightly closer on average. Given the close performance, we opt for recommending t = 0.005 as higher values are less likely to overfit on the validation set (bias-variance tradeoff). In the limited evaluation results on Table 5 we compare the performance of conventional layer choices against WSD sense profiles with the recommended temperature value. We observe that for 11 out of 14 models, 6 t = 0.001 results in large exponents that cause overflow errors. WSD sense profiles with the recommended temperature reliably outperform any of the conventional choices, of which none stands out as a reliable cross-model choice. Moreover, on Table 5 we also see that WSD sense profiles with the recommended temperature generally match or outperform both the single layers which performed best on the validation set, and the WSD sense profiles using the temperature value that showed best performance on the validation set.
Our findings regarding performance of USM sense profiles largely follow the previously mentioned findings for WSD sense profiles. In the case of USM, validation results on Table 6 more clearly show that t = 0.100 performs better, although t = 1.000 also performs well. As for the limited evaluation results, Table 7 shows that conventional layer choices significantly underperform any of the alternatives introduced in this work, with the USM sense profile with recommended temperature (t = 0.100) showing overall best performance.

Choosing Sense Profiles for Different Tasks
Having established that our proposed sense profiles improve WSD and USM performance over conventional layer choices, the question remains of whether to choose WSD or USM sense profiles to represent sense embeddings. In this work we propose choosing sense profiles based on the probing task that shares most similar constraints to the downstream task of interest. More specifically, tasks requiring comparison of different senses for the same word fit the disambiguation profile, such as classical WSD (Navigli, 2009) or WiC (Pilehvar & Camacho-Collados, 2019), and benefit less from information in lower layers. On the other hand, tasks without lexical constraints, not only USM but also synset similarity (Colla et al., 2020a) or semantic change (Hamilton et al., 2016), are better suited to the matching profile, which uses information from more layers. In Section 7 we evaluate sense embeddings learned using sense profiles according to each task's constraints, and in Section 8.1 we analyse the performance gap when using alternate sense profiles.

Evaluation
In this work we address several sense-related tasks selected to investigate the versatility of the proposed sense embeddings, covering disambiguation (  All tasks are solved essentially using cosine similarity between contextual embeddings and LMMS-SP precomputed sense embeddings represented using the same NLM. Each task's subsection provides more details about how these similarities are used to produce task-specific predictions. No additional taskspecific training or validation datasets are used asides from those referred in Section 5.2, and all NLMs are employed in the same exact fashion -simply retrieving contextualized representations from each layer (following §4.1).
As such, LMMS-SP performance on these tasks should be indicative of each NLM's intrinsic ability to approximate meaning representations learned during pre-training with language modelling objectives alone.

Sentence Lemma POS Gold Sensekey
Eyes that were clear , but also bright with a strange intensity , a sort of cold fire burning behind them . fire NOUN fire%1:12:00:: WSD is the most popular and obvious task for evaluating sense embeddings.
This task has been researched since the early days of Artificial Intelligence and constitutes an AI-complete task (Navigli, 2009). It is usually formulated as choosing the correct sense for a word in context out of a list of possible senses given the word's lemma and part-of-speech tag (see Table 8). Several test sets have been proposed over the years, and the compilation of Raganato et al. (2017) has emerged as the de facto evaluation framework for English WSD, which we also use. Naturally, this task suits sense profiles for WSD, and we follow the method described in Section 4.3.

Results
On Table 9 we report performance on the standard test sets of the WSD Evaluation Framework (Raganato et al., 2017). Given the breadth of recent As could be expected, task-specific classifiers show best results, particularly BEM (Blevins & Zettlemoyer, 2020) and ConSeC (Barba et al., 2021 (Raganato et al., 2017). Top rows show results for models using sense annotations exclusively from SemCor (SC). Bottom rows show results for models augmenting SC with annotations from additional sources. Results marked with * correspond to development sets (and therefore ALL). For each group of results, we underline the best from LMMS. 7 We expect the annotations in ARES to produce further performance gains for LMMS-SP but do not use this resource due to its large size (13x the annotations in SemCor+UWA10).

Sentence Gold Sensekey Gold Synset
Eyes that were clear , but also bright with a strange intensity , a sort of cold fire burning behind them . fire%1:12:00:: 06711159n (fire 9 n )  Table 8 and Table 10). Conveniently, this allows for USM to use the same test sets as WSD. As expected, we address USM using the sense profile of the same name, and follow the method described in Section 4.3. In this work we evaluate USM from both the sensekey and synset perspective, to provide a clearer account of the impact of lexical information on task performance. (2020), we evaluate performance considering two additional metrics besides F1: Precision at 5 (P@5) and Mean

Following Loureiro & Camacho-Collados
Reciprocal Rank (MRR). To our knowledge, ARES (Scarlini et al., 2020b) is the only other publicly available set of full-coverage sense embeddings represented in the space of a Transformer-based NLM, so we also compare LMMS-SP against those sense embeddings. Since our prior LMMS sense embeddings and the ARES sense embeddings are released using sensekey representations, USM synset evaluation requires converting those sensekey embeddings to synset embeddings.
We perform this conversion by simply averaging sensekey embeddings that belong to the same synset. In Section 8.2 we analyse the impact this conversion can have on task performance. On

Lemma POS Boolean
You must carry your camping gear .
Sound carries well over water .

carry VERB False
He wore a jock strap with a metal cup .
Bees filled the waxen cups with honey .

Word-in-Context (WiC)
The Word-in-Context (Pilehvar & Camacho-Collados, 2019, WiC) task is designed to assess how context impacts word representations produced by contextual NLMs. It is a binary classification task that simply requires determining whether a particular word is used with the same meaning or not in a pair of sentences, also given lemma and POS provided in WSD tasks (see Table 12 for examples). The dataset is balanced and performance is measured with accuracy.

Solution
In this work, we tackle the WiC task using our proposed sense embeddings following the unsupervised approach from Loureiro & Jorge (2019b), which essentially applies the 1NN method for disambiguating the target word in both sentences and checks whether they are equal or not. Even though we also explored a supervised approach in Loureiro & Jorge (2019b), based on Logistic Regression, in this work we focus on the unsupervised approach as its performance is more revealing of the inherent representational abilities of NLMs.
Given the close relation to disambiguation, we use WSD sense profiles for WiC.

Results
WiC is a benchmark NLU task, being part of SuperGLUE (Wang et al., 2019), therefore most state-of-the-art NLMs have reported results for this task.
The initial baseline methods proposed with WiC were based on cosine similarity with thresholds learned from the validation set.
Most recent solutions, however, involve fine-tuning the NLM (as performed for other sentence classification tasks in SuperGLUE) using the training and validation sets provided with WiC. One notable exception is Scarlini et al. (2020b) which proposed a method that leverages ARES sense embeddings to improve the fine-tuning process. As such, on Table 13 we compare results from these solutions to our unsupervised LMMS and LMMS-SP, as well as an unsupervised result based on the same 1NN approach using the ARES embeddings.
Starting with our unsupervised results, we confirm that LMMS-SP BERT-L surpasses the performance of LMMS 2048 (based on BERT-L), and once again LMMS-SP ALBERT-XXL displays the best performance. Nevertheless, supervised solutions using NLMs fine-tuned for this task show best performance overall, particularly T5 (Raffel et al., 2020) which is currently the largest NLM with reported results on this task, at over 11B parameters. KnowBERT (Peters et al., 2019) and SenseBERT (Levine et al., 2020)  The method used by Scarlini et al. (2020b) to employ sense embeddings while fine-tuning BERT-L for WiC resulted in a notable improvement similar to Sense-BERT. In the unsupervised setting, however, we found that ARES embeddings outperform LMMS-SP BERT-L , but underperform both LMMS-SP RoBERTa-L and LMMS-SP ALBERT-XXL . We expect following the same method to assist supervised fine-tuning with LMMS-SP sense embeddings may produce improved results, but consider that experiment out of scope for this work.
As for solutions using the threshold method, all reported models substantially underperform unsupervised results using any Transformer-based NLM.

Graded Word Similarity in Context (GWCS)
For evaluating graded contextual similarity, in contrast to the binary contextual similarity assignments of WiC, we address SemEval 2020 Task 3: Graded Word Similarity in Context (GWCS) (Armendariz et al., 2020a). This task,

Method
Language Model Sense Embeddings Acc.

72.1
Fine-Tuning T5 (2020) -  based on the CoSimLex resource (Armendariz et al., 2020b), targets word pairs used for evaluating distributional semantic models (not necessarily polysemous words) in contexts spanning multiple sentences. The task is divided into two sub-tasks derived from human-annotated similarity ratings: 1) predict the change in similarity between two different contexts for each word pair; 2) predict the similarity ratings themselves. Table 14 shows a single example from GWCS, featuring two contexts each with occurrences of the same pair of words, context specific similarity ratings, and the associated similarity change.

Contexts
Sim. Change

Solution
While the sub-tasks are independently evaluated, we employ essentially the same method for both, based on our straightforward approach for the WiC task covered in Section 7.3, with minor adjustments to quantify the observed change in similarity. Given contexts A and B, we disambiguate target words (each instance's word pair) in the corresponding contexts, and compute sense similarities sim A wsd and sim B wsd as the cosine similarity between the embeddings of the predicted senses. Considering that disambiguation may predict the same senses, thus resulting in sim A wsd = sim B wsd for many instances, we also compute contextual similarities sim A ctx and sim B ctx as the cosine similarity between the contextual embeddings of the target words. Thus, we determine similarity scores specific to context A as sim A = 1 2 (sim A wsd + sim A ctx ), and similarity scores specific to context B as sim B = 1 2 (sim B wsd + sim B ctx ). These context-specific similarities constitute our solutions to sub-task 2. We determine the semantic change scores for sub-task 1 trivially as sim B − sim A . Considering that this solution closely follows our solution for WiC, and that word pairs contained in this dataset tend to be closely related, we use the WSD sense profile for GWCS.

Results
Performance on sub-task 1 is measured with Pearson Uncentered Correlation between the system's scores and the average human annotations, and perfor-mance on sub-task 2 is measured with the harmonic mean of the Spearman and Pearson correlations between the system's scores and the average human annotations. On Table 15 we report results using our sense embeddings (LMMS and LMMS-SP), using the ARES sense embeddings with our scoring method (and BERT-Large, pooling with the sum of last 4 layers), and the best reported results from other task participants (including post-evaluation, until 03/2021).
Similarly to WiC, the scores for the test sets are hidden from participants, both during evaluation (ended 03/2020) and post-evaluation periods (extends indefinitely), so all reported results are obtained from the online platform used by SemEval after submitting each system's predictions.
We observe that our straightforward method combining similarity between sense and contextual embeddings is able to outperform the solutions of other task participants (Leaderboard Best), most of which also relied on Transformerbased NLMs (Armendariz et al., 2020a). Interestingly, GWCS shows wide variation in performance from the choice of NLM, with LMMS-SP XLNet-L standing out with clearly best results on both sub-tasks 8 .
Showing synset identifiers after conversion to WordNet (more readable format in parenthesis).
All the tasks we considered so far (WSD, USM, WiC and GWCS) have evaluated sense embeddings by their utility for accurately matching or distinguishing word senses in particular contexts. In this last task, we address intrinsic evaluation of sense embeddings, directly comparing distributional similarity between sense pairs against human similarity ratings.
We perform this evaluation using the Sense Identification Dataset (Colla et al., 2020c, SID), which is based on the word pairs (nouns only) and human similarity ratings from SemEval-2017 Task Table 16).

Task Adaptation
We convert the BabelNet sense identifiers to synsets from WordNet 3.0 using the mapping provided by Navigli & Ponzetto (2010). However, some instances cannot be mapped due to missing entries in WordNet, or, in rare cases, their mapping results in the two senses of the pair being equal, leading to a reduction of 492 instances to 377 mapped to WordNet. We further split SID into different groups for additional insights. We first separate the 354 pairs for which both senses are represented in the related works we compare against (overlapping), considering these are not always complete sets of WordNet sense embeddings.
Next, we breakdown the overlapping pairs into a set of the most polarized word pairs (i.e., similarity ratings ≤ 1 or ≥ 3), and another set containing only pairs with senses that are annotated in SemCor+UWA10 (observed).

Solution
We use cosine similarity between synset embeddings to correlate with human similarity ratings. Since we are directly comparing embeddings of very different dimensionality, we apply truncated SVD to normalize them to 300 dimensions 9 (including related work). The senses being compared range from completely unrelated (e.g., polyhedron 1 n ; actor 1 n ) to highly related or similar (e.g., actor 1 n ; actress 1 n ), so we use USM sense profiles for SID.

Results
Performance on SID is measured with Pearson correlation. For completeness, we report performance of synset embeddings that are not based on con- Results for the WordNet-subset of SID are shown on Table 17. As can be observed, LMMS-SP substantially outperforms LMMS and related works. As with GWCS, LMMS-SP XLNet-L stands out with clearly best results. While LMMS-SP also outperforms most non-contextual embeddings, it still underperforms LessLex (Colla et al., 2020b) embeddings, which are based on ensembles and learned using BabelNet. We also note that LMMS-SP performs particularly well on the 'Observed' set corresponding to senses learned from annotated corpora. The performance gap between ARES and LMMS-SP BERT-L suggests that additional semi-supervised annotations for more senses may not suffice. Finally, the 'Polarized' set seems consistently easier than the full set, indicating that the most challenging pairs are those with moderate similarity ratings.

Analysis
In this section, we perform several ablation studies to better understand the 11 Due to leaderboard submission limits, ablations for WiC use the validation set.

Choice of Sense Profiles
On Table 18 we report performance according to the sense profile used for weighted pooling of contextual embeddings from NLMs (described in Section 6.2), and using the sum of the last 4 layers (Sum-LST4), as commonly used in related work and the original LMMS (Loureiro & Jorge, 2019a).
Our first conclusion is that Sum-LST4 pooling is only appropriate for particular models and tasks (i.e., WSD and WiC w/BERT-L; WiC w/RoBERTa-L), but detrimental for most (specially any task w/XLNet-L; any model for USM).
However, our recommended choice of sense profile not only appears beneficial for WSD and USM tasks across all models (expected since the sense profiles are based on those tasks), but also for WiC, GWCS and SID. In fact, out of 20 model-task combinations, we only find 3 exceptions: RoBERTa-L on WiC and GWCS, and ALBERT-XXL on GWCS (to a lesser extent). Moreover, we confirm that tasks are sensitive to the choice between SP-WSD and SP-USM, which validate our task-specific recommendations.

Unambiguous Word Annotations
In this work we learnt our initial set of sense embeddings (as described in Sections 4.1 and 4.2.1) using SemCor, the only source of sense annotations used for LMMS (2019a), in combination with UWA (Loureiro & Camacho-Collados, 2020), a set of sense annotations exclusively targeting unambiguous words.
On Table 19 we present results showing the impact of UWA on task performance. As noted in Loureiro & Camacho-Collados (2020), the increase in Word-Net coverage using UWA allows for disentangling dense clusters that coarsen the semantic space when relying on SemCor and network propagation alone. Consequently, we expect UWA to benefit sense matching tasks, which is confirmed by our results showing substantial improvements in USM and SID (the two tasks using SP-USM). We also find that UWA does not hinder performance on the remaining tasks for most model-task combinations (improves in most cases), with the exceptions of GWCS with RoBERTa-L and WiC with ALBERT-XXL (the former is also an exception observed in the sense profile ablation on §8.1).
As future work, we will also explore WordNet-independent procedures to discover monosemous words, such as the method introduced by Soler & Apidianaki (2021), which may lead to further improvements.

Merging Gloss Representations
Another aspect of LMMS-SP that differs from LMMS (2019a) is merging gloss embeddings by averaging with sense embeddings, instead of through concatenation (described in Section 4.2.2).
Results on Table 20 reveal that concatenation only benefits WSD, with minor improvements over averaging. Alternatively, averaging shows clear improvements for all other tasks (again, the exception is GWCS w/RoBERTa-L).
We also report performance using exclusively gloss representations, and sense embeddings without gloss information.

Learning Synsets Directly
The SID task, as well as the synset version of USM, require synset-level embeddings. In Section 4.1, we explain that LMMS-SP synset embeddings are learned directly from sensekey annotations that are converted to synsets. However, in our evaluation we compare LMMS-SP with other works that are only available as sensekey embeddings, so we converted these representations into synset embeddings learned as the average of corresponding sensekey embeddings (i.e., learned indirectly). On

Discussion
In this section we discuss the main findings of our work. More specifically, we discuss sense representation at specific layers of NLMs ( §9.1), differences observed across models and variants ( §9.2), and finally, how our sense embeddings may benefit downstream tasks ( §9.3).

Layer Distribution
Throughout this article we have provided empirical evidence supporting that there is substantial non-monotonic variation in the adeptness of specific layers of Transformer-based NLMs for sense representation. This evidence is available from both our probing analysis and the improvements in several sense-related tasks obtained from using our proposed sense profiles, which are based on nonmonotonic pooling from all layers (most clearly shown in Table 3).
The cause for this variation remains elusive, calling for controlled experiments where different NLMs are tested under comparable circumstances, particularly with regards to training data and modelling objectives, although such an experimental setup may be cost-prohibitive for models of this scale. Nevertheless, seeking to better understand this variation, we conducted two qualitative experiments targeting representations of the same sentences at different layers.
In our first qualitative experiment, we compared sense similarity at different layers for the same word in context. We found some evidence potentially in support of the hypothesis advanced by Voita et al. (2019b), with the distribution of final layers resembling the distribution of the first layers moreso than the distribution of middlemost layers, where the difference between correct and incorrect senses is more marked (see example in Figure 2). to the shape, public square (correct, green bar chart) and mathematical operation, in order.
Initial layer similarities are influenced by the word forms used with each sense in SemCor.
We further extended the previous experiment to a cluster-level comparison of the embedding space. For this experiment, we focus on the words present in the CoarseWSD-20 dataset (Loureiro et al., 2021), both in aggregate for measuring correct sense clustering, as well as targeting "spring" and its three distinct senses for visualization. Considering silhouette scores 12 (Rousseeuw, 1987) and PCA visualizations of the embedding space (  of CoarseWSD-20 (Loureiro et al., 2021). Top rows report scores for specific layers, bottom rows report scores when pooling the sum of last 4 layers and our proposed pooling strategies. 12 We use the mean silhouette coefficients of all embeddings for a particular word to measure how well each model and pooling strategy can assign embeddings to the correct sense cluster.
Silhouette coefficients are based on intra-and nearest-cluster cosine similarities. Low values represent overlapping clusters. CoarseWSD-20 (Loureiro et al., 2021). Using PCA for dimensionality reduction. Silhouette scores s are computed before reduction.

NLM Idiosyncrasies
Besides unexpected results regarding the performance of particular layers of NLMs, we also find intriguing differences in the patterns of layer performance observed across models, and even variants of the same model. Looking at our results on Table 3, we find many intriguing examples of this variation. The reasons for these differences in patterns across models and variants are not straightforward, specially considering many of these models are trained on very similar data and architectures. Still, among several technical differences, we highlight the differences in modelling objectives covered in Section 5.1. Out of the 4 the models we considered in this work (see performance summary on Table 24), XLNet is in fact simultaneously the model that appears most distinctive, with particularly strong performance on graded similarity tasks, and whose objectives are most different (being the only model not using MLM). Another interesting finding is that we obtain best results on WSD, USM and WiC using ALBERT-XXL, which has half the layers of the other models, but much larger embedding dimensionality (model details are available in Table 2). As for differences in variants of the same model (same objectives) we consider the possibility that trivial run-time parameters may have an impact on this variation, akin to the unexpected influence of random seeds on fine-tuning BERT models (Dodge et al., 2020).

Knowledge Integration
The ability of matching WordNet synsets to any fragment of text allows downstream applications to easily leverage the manually curated relations available on WordNet. At the same time, these sense embeddings can also serve as an entry point to many other knowledge bases linked to WordNet, such as the multilingual knowledge graph of BabelNet (Navigli & Ponzetto, 2010), the common-sense triples of ConceptNet (Speer et al., 2017) or WebChild (Tandon et al., 2017), the semantic frames of VerbNet (Schuler, 2006), and even the images of ImageNet (Russakovsky et al., 2015) or Visual Genome (Krishna et al., 2016). Several recent works have used the symbolic relations expressed in these knowledge bases to improve neural solutions to Natural Language Inference (Kapanipathi et al., 2020), Commonsense Reasoning (Lin et al., 2019), Story Generation (Ammanabrolu et al., 2020), among others.
As an example of how using LMMS-SP to bridge natural language and symbolic knowledge can be beneficial, in Figure 4 we demonstrate how these sense embeddings allow for generalization of argument spans, predicted by a semantic parser, exploiting WordNet relations between matched synsets. The matches shown in Figure 4 also illustrate how sense embeddings may be used for probing world knowledge encoded in pre-trained NLMs, as already suggested in Loureiro & Jorge (2019a).

Conclusion
Leveraging neural language models in combination with sense-annotated corpora (and complementary resources such as glosses or relations), this work has shown that it is possible to produce sense embeddings applicable beyond mere disambiguation, with relevant implications for long-standing challenges in Artificial Intelligence such as symbol grounding.
This extension of Loureiro & Jorge (2019a) proposes a more principled approach for learning distributional representations of word senses using pretrained NLMs, focusing on state-of-the-art Transformer-based models. From extensive evaluation on several sense-related tasks, we demonstrated that the LMMS-SP approach is more effective than prior work at approximating precise word sense representations in the same vector space of NLMs.
The broad probing analysis of the many variants of popular NLMs endeavored in this work provides new evidence supporting further research on the interplay between pre-training objectives, layer specialization, and model size.
The conclusions of this probing analysis are indeed expected to be applicable in tasks outside WSD, and for learning representations other than sense embeddings, which we leave for future work.
Effectively, there are known limitations to meaning representation based on language modelling objectives alone (Bender & Koller, 2020;Merrill et al., 2021). Nonetheless, we believe our work shows there is still much to understand about how to best leverage NLMs for meaning representation, in addition to more thoroughly testing the effectiveness of current approaches centered on self-supervision.
Release. This work is accompanied by the release of the following resources: sensekey and synset embeddings with full-coverage of WordNet based on BERT-L, XLNet-L, RoBERTa-L and ALBERT-XXL; scripts to generate embeddings following our method, using the same NLMs or others supported by the Transformers package; and scripts to run task evaluations. These resources are released under a GNU General Public License (v3) and available from this public repository: https://github.com/danlou/lmms

Acknowledgements
Daniel Loureiro is supported by the EU and Fundação para a Ciência e Tecnologia through contract DFA/BD/9028/2020 (Programa Operacional Regional Norte). Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship.

Appendices
A. Full results for SemEval 2020 -Task 3 On

B. Stanford Contextual Word Similarities (SCWS)
On Table 26 we report our results on the Stanford Contextual Word Similarities (Huang et al., 2012, SCWS) task. We address this task similarly to GWCS (see Section 7.4). Given two words in context, each within an independent sentence, we disambiguate both occurrences and score each pair as the average of similarities between corresponding sense and contextual embeddings.
Results on SCWS follow performance on GWCS, with XLNet-L outperforming other NLMs as well as results from related works. Analysing performance by Part-of-Speech (POS), we find that nouns appear most challenging for this task, particularly when being compared against other nouns.   pre-training for natural language generation, translation, and comprehension.