We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context. For writing, lexical substitution systems can assist humans by suggesting words that humans cannot easily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to humans. Furthermore, annotators often provide substitutes of low quality, which are not actually appropriate in the given context. We collect higher-coverage and higher-quality data by framing lexical substitution as a classification problem, guided by the intuition that it is easier for humans to judge the appropriateness of candidate substitutes than conjure them from memory. To this end, we use a context-free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our SWORDS benchmark has 3 x as many substitutes per target word for the same level of quality, and its substitutes are 1 . 4 x more appropriate (based on human judgement) for the same number of substitutes. Imagine you are writing the message I read an amazing paper today to a colleague, but you want to choose a more descriptive adjective to replace amazing. At first you might think of substitutes like awesome and great, but feel that these are also unsatisfactory. You turn to a thesaurus for inspiration, but among reasonable alternatives like incredible and fascinating are words like prodigious which do not quite fit in your context. Ultimately, you choose to go with fascinating, but reaching this decision required a non-trivial amount of time and effort. Research on lexical substitution (McCarthy, 2002; McCarthy and Navigli, 2007; Erk and Pad, 2008; Szarvas et al., 2013; Kremer et al., 2014; Melamud et al., 2015; Hintz and Biemann, 2016; Zhou et al., 2019; Arefyev et al., 2020) considers the task of replacing a target word in context with appropriate substitutes. There are two widely-used English benchmarks for this task: SEMEVAL (McCarthy and Navigli, 2007) and COINCO (Kremer et al., 2014). For both benchmarks, data was collected by asking human annotators to think of substitutes from memory. Because lexical substitution was originally proposed as a means for evaluating word sense disambiguation systems (McCarthy, 2002), this data collection strategy was designed to avoid a bias towards any particular word sense inventory. In this work, we consider a different use case for lexical substitution: writing assistance. For this use case, we are interested in evaluating a system's ability to produce appropriate substitutes that are likely to be difficult for humans to think of. We show that the data collection strategy used in past benchmarks yields low coverage of such uncommon substitutesfor our previous example, they might contain words like awesome and great, but miss words like incredible and fascinating. Furthermore, we observe that these benchmarks have low quality , containing words like fun, which are easy to think of, but not quite appropriate in context. We present SWORDS the Stanford Word Substitution Benchmarkan English lexical substitution benchmark that raises the bar for both coverage and quality (Table 1). We collect SWORDS by asking human annotators to judge whether a given candidate word is an appropriate substitute for a target word in context, following the intuition that judging a given substitute is easier than producing that same substitute from memory. To bootstrap a set of candidates for humans to annotate, we Context My favorite thing about her is her straightforward honesty. look up target words in an existing context-free thesaurus. 1 Because a thesaurus might miss substitutes that would not typically be synonymous with the target word outside of the provided context (e.g. thought-provoking for amazing), we also include human-proposed candidates from the previous COINCO benchmark. Determining whether a substitute is appropriate is intrinsically subjective. To address this, we collect binary labels from up to ten annotators for each substitute, inducing a score for each substitute. In COINCO , analogous scores are derived from the number of independent annotators who thought of a substitutehence, as we will show in Section 4, these scores tend to correspond more to ease-of-recollection than appropriateness. In contrast, scores from SWORDS correspond to appropriateness, and also allow us to explicitly trade off coverage and quality, permitting more nuanced evaluation. Our analysis shows that compared to COINCO , SWORDS has 3 x more substitutes per target word for the same level of quality, and its substitutes are 1 . 4 x more appropriate based on scores for the same number of substitutes. We demonstrate that SWORDS is a challenging benchmark by evaluating state-of-the-art lexical substitution systems and large-scale, pre-trained language models including systems based on BERT (Devlin et al., 2019; Zhou et al., 2019) and GPT-3 (Brown et al., 2020). In our evaluation, we find 1 Note that our use of a thesaurus makes SWORDS less appropriate for the original use case for lexical substitution: evaluating word sense disambiguation systems. that humans substantially outperform all existing systems, suggesting that lexical substitution can be used as a downstream language understanding task for pre-trained models. We release SWORDS publicly as a benchmark for lexical substitution, coupled with a Python library that includes previous benchmarks in a common format, standardized evaluation scripts for prescribed metrics, and reproducible re-implementations of several baselines. 2 2 Background We describe lexical substitution and briefly introduce two widely-used benchmarks: SEMEVAL (McCarthy and Navigli, 2007), the first benchmark, and COINCO (Kremer et al., 2014), the largest existing benchmark. For a survey of other benchmarks, we refer readers to Kremer et al. (2014), Hintz and Biemann (2016), and Miller (2016). Lexical substitution. Lexical substitution is the task of generating a list of substitutes w (cid:48) that can replace a given target word w in a given context c (McCarthy, 2002): ( context c, target w )  [ substitute w (cid:48) ] . The context c is one or more sentences where the target word w is situated. The target word w is one word in the context, which is either manually chosen by humans (McCarthy and Navigli, 2007) or 2 SWORDS : github.com/p-lambda/swords All experiments reproducible on the CodaLab platform: worksheets.codalab.org/worksheets/0xc924392d555f4b4fbee47be92e3daa0b automatically selected based on the part-of-speech of the target word (Kremer et al., 2014). The substitute w (cid:48) can be a word or phrase. Note that the task of lexical substitution does not consider in-flection and does not involve grammar correction; all benchmarks contain lemmas as substitutes (e.g. run instead of ran). SEMEVAL . The first lexical substitution benchmark, SEMEVAL -2007 Task 10 (McCarthy and Navigli, 2007), contains 201 manually chosen target words. For each target word, 10 sentences were chosen as contexts (mostly at random, but in part by hand) from the English Internet Corpus (Sharoff, 2006) and presented to five human annotators. The five annotators were instructed to produce up to three substitutes from memory as a replacement for the target word in context that preserves the meaning of the original word. This resulted in 12 , 300 labels in total with four substitutes per target word on average. COINCO . The previous largest lexical substitution benchmark, COINCO (Kremer et al., 2014), was constructed by first choosing 2474 contexts from the Manually Annotated Sub-Corpus (Ide et al., 2008, 2010). Then, all content words (nouns, verbs, adjective, and adverbs) in the sentences were selected to be target words in order to reflect a realistic frequency distribution of target words and their senses. Each target word was presented to six human annotators, who were asked to provide up to five substitutions or mark it as unsubstitutable. All the annotators were instructed to provide (prefer-ably single-word) substitutes for the target that would not change the meaning. This resulted in 167 , 446 labels in total and 7 . 2 substitutions per target word on average. 3 For the rest of the paper, we focus on COINCO (but not SEMEVAL ) as our benchmark is built on COINCO and it is the largest existing benchmark. SWORDS is composed of context, target word, and substitute triples ( c, w, w (cid:48) ), each of which has a score that indicates the appropriateness of the substitute. We consider a substitute to be acceptable if its score is greater than 50% (e.g. bolded words in Table 1) and unacceptable if the score is less than 3 The reported number in Kremer et al. (2014) is 167 , 336 and 10 . 71 , respectively. The latter differs as they counted the same substitute multiple times when suggested by multiple humans, whereas we report the number of unique substitutes. or equal to 50%. Similarly, a substitute with a score greater than 0% is considered conceivable , and otherwise inconceivable . Note that these terms are operational definitions for convenience, and different thresholds can be chosen for desired applications. Improving quality. In prior work, annotators were prompted to consider whether a substitute preserves the meaning (McCarthy and Navigli, 2007) or would not change the meaning (Kremer et al., 2014) of the target word. Instead, we ask annotators whether they would actually consider using this substitute as the author of the original sentence. We believe this wording encourages a higher standard. In Section 4.1, we provide evidence that substitutes from SWORDS have higher quality than those from COINCO on average. Improving coverage. For prior benchmarks, annotators were asked to generate a list of substitutes from memory. Psycholinguistic studies have shown that when humans are asked to predict the next word of a sentence, they deviate systematically from the true corpus probabilities (Smith and Levy, 2011; Eisape et al., 2020). Thus, we may reasonably expect that asking humans to generate substitutes would similarly lead to systematic omissions of some appropriate substitutes. We observe that prior benchmarks exclude many appropriate substitutes that are difficult for humans to think of (Section 4.2). To address this limitation, we first obtain a set of candidate substitutes and then ask annotators to judge whether they would consider using a given candidate to replace the target word in the context. That is, given a context c , target word w , and candidate substitute w (cid:48) , we ask humans to judge whether w (cid:48) is a good replacement for the target word: ( context c, target w, substitute w (cid:48) )  { 0 , 1 } , where a positive label 1 corresponds to I would actually consider using this substitute as the author of the original sentence, and a negative label 0 as the opposite. As described in Section 3.2, we annotate a large pool of candidate substitutes to ensure high coverage of all possible substitutes. We confirm that this increases coverage compared to COINCO in Section 4.2. score defined as the number of annotators who produced w (cid:48) given the associated context c and target word w . Instead, we define the score as the fraction of annotators who judged the w (cid:48) to be an appropriate replacement of w . We argue that the previous definition of score reflects ease-of-recollection, but not necessarily appropriateness. In Section 4.3, we show that our definition of score better represents the appropriateness of each substitute. We collect substitutes and scores for a context and target word pair ( c, w ) via the following three steps. Step 1: Select contexts, targets, and substitutes. We use the subset of contexts and target words from COINCO . Concretely, we start with the ( c, w ) pairs in COINCO and randomly select one w per c to annotate. Here, the context c consists of three sentences, where the middle sentence has the target word w . Next, we choose a set of candidate substitutes w (cid:48) to annotate for each ( c, w ) pair, as framing annotation as binary classification requires determining the set of candidate substitutes a priori. We use human-generated substitutes from COINCO , then add substitutes suggested by a thesaurus (see Appendix A.2 for details). In principle, candidate substitutes can be retrieved from any lexical resource or even sampled from a generative model, which we leave as future work. By combining candidates from COINCO and the thesaurus, we increase the coverage of acceptable substitutes. a list of candidate substitutes from the previous step, we collect three binary labels on each ( c, w, w (cid:48) ) triple (see Section 3.3 for details). Then, we pass any substitute with at least one positive label to Step 3 and further collect fine-grained scores. We show that the probability that an acceptable substitute gets incorrectly filtered out as an inconceivable substitute (three negative labels) is very low ( 0 . 8 %) in Section 4.4. Step 3: Collect fine-grained scores. In the final step, we collect seven more binary labels on the substitutes which received at least one positive label from Step 2. This yields a total of 10 binary labels for the substitutes. We used Amazon Mechanical Turk (AMT) to crowdsource labels on substitutes. Each Human Intelligence Task (HIT) contained a target word highlighted in the context and at most 10 candidate substitutes for the target word. Each candidate substitute had three radio buttons for positive, negative, and abstain. Annotators were asked to choose positive if they would actually consider using the substitute to replace the target word as the author of the context, negative if they would not consider using the substitute, and abstain if they do not know the meaning of the substitute. We treated all abstain labels ( 1 . 24 % of total labels) as negative labels, thereby making it binary. The benchmark includes abstain labels to maintain the option for them to be handled separately (e.g. excluded) in the future. The interface, instructions, qualification conditions, and filtering criteria used for crowdsourcing can be found in Appendix B. 4 Data analysis Table 2 shows overall statistics of our benchmark. SWORDS comprises a total of 1250 context and target word pairs ( 494 nouns, 448 verbs, 189 adjectives, 119 adverbs) and 71 , 813 total substitutes that have been labeled (including both acceptable and unacceptable substitutes). For brevity, we defer an analysis of annotator agreement to Appendix C.1. With our notion of acceptability, we first observe that 75 . 4 % of the substitutes from COINCO 4 are considered unacceptable, and 28 . 6 % of the substitutes are even inconceivable (receiving scores less than 50 % and 0 % from our human annotators). Table 3 shows examples of substitutes that received relatively high scores under COINCO , yet were considered unacceptable under SWORDS . With the same size as COINCO (by taking the subset of our benchmark with the highest scoring substitutes per target), the average score of the substitutes is 4 . 9 for SWORDS and 3 . 4 for COINCO , resulting in 1 . 4 x higher quality. Furthermore, SWORDS minimizes the potential noise by having fine-grained scores to account for appropriateness (Section 4.3) as well as explicit inconceivable substitutes, which is useful for evaluation (Section 5.2). We show that SWORDS achieves high coverage. Among the conceivable substitutes in SWORDS , 14 . 4 % are only in COINCO (COINCO -only), 14 . 6 % are common to both COINCO and the thesaurus (COINCO  Thesaurus), and 71 . 1 % are only from thesaurus (Thesaurus-only). Among the acceptable substitutes, 24 % are from COINCO -only, 37 . 1 % are from COINCO  Thesaurus, and 38 . 9 % are from Thesaurus-only. This suggests that a substantial number of substitutes are not present in COINCO . Overall, SWORDS contains 3 . 9 acceptable and 20 . 1 conceivable substitutes per target word on average, increasing those numbers by nearly 2x and 3x over COINCO , respectively. In addition, we find that substitutes from COINCO -only are more likely to be common words whereas substitutes from Thesaurus-only are more likely to be rare words. We compute the Zipf frequency (Speer et al., 2018) of each substitute based on the Google n -gram corpus (Brants and Franz, 4 For this analysis, we consider COINCO 's substitutes that are used and labeled under SWORDS . 2006) and threshold conceivable substitutes into three groups: uncommon (  3 . 5 ), neutral, common ( > 4 . 5 ). We observe that substitutes from COINCO -only are more likely to be common words ( 52 . 7 %) than those from Thesaurus-only ( 38 %). On the other hand, the substitutes from Thesaurus-only tend to be more uncommon words ( 29 %) than those from COINCO -only ( 17 . 5 %). We show that scores in SWORDS better reflect the appropriateness of each substitute compared to COINCO both quantitatively and qualitatively. We find that if a substitute has a high score under COINCO (score > 1 ), it is likely to be acceptable under SWORDS (score > 50 %) almost all the time ( 99 . 6 %). However, the converse does not hold: the acceptable substitutes under SWORDS have low scores (score  1 ) under COINCO half of the time ( 49 . 4 %). Intuitively, this is because COINCO 's scores reflect the ease of producing the substitute from memory, whereas SWORDS 's scores reflect the appropriateness of the substitute. Table 3 shows examples of context, target word, and substitute triples which received a low score from COINCO but a high score from SWORDS . We show that the probability of an acceptable substitute falsely filtered out in Step 2 is very low. To this end, we collected 10 additional labels on 100 context-target word pairs randomly selected from the test set, without reducing the pool of substitutes as in Step 2. By comparing the first three labels to the entire 10 labels, we find that 35.5% of substitutes without any positive labels in Step 2 could have received one or more positive labels if they were kept in Step 3. However, we find that 99 . 2 % of these substitutes were eventually considered unacceptable (judged by 10 labels), indicating that the probability of an acceptable substitute incorrectly filtered out in Step 2 is very low ( 0 . 8 %). Figure 1 shows the score distribution of substitutes in SWORDS along with the source of substitutes: COINCO -only, COINCO  Thesaurus, or Thesaurus-only. Across scores, neither COINCO nor thesaurus completely dominates substitutes, and the overlap between COINCO and thesaurus is 5 1935 substitutes from COINCO -only, 972 from both, and 43 , 806 from thesaurus-only received a score of 0%. We also find that SWORDS adds more substitutes for all the scores, although substitutes from the thesaurus tend to have a lower range of scores compared to those from COINCO . Lastly, we observe that substitutes from COINCO roughly form a normal distribution, which suggests that even the substitutes provided by human annotators are controvertible, and that it is important to account for the intrinsically gradable nature of appropriateness with fine-grained scores. In this section, we evaluate several methods on SWORDS . The goals of this evaluation are threefold: (1) to prescribe our recommended evaluation practice for SWORDS , (2) to measure performance of existing large-scale pre-trained models and state-of-the-art lexical substitution systems, and (3) to measure human performance for the purpose of comparing current and future systems. There are two primary evaluation settings in lexical substitution research: the generative setting (Mc-Carthy and Navigli, 2007) and the ranking setting (Thater et al., 2010). In the generative setting, systems output a ranked list of candidate substitutes. There are no restrictions on the number of candidates that a system may output. In the ranking setting, systems are given all candidate substitutes from the benchmark (including those marked as unacceptable) and tasked with ranking them by appropriateness. Here we primarily focus on the generative setting, as it is more relevant to writing assistance. We defer our experiments on the ranking setting to Appendix D. 5.2 Evaluation metrics In a writing assistance context, we envision that lexical substitution systems would be used to suggest a limited number of substitutes to users (e.g. 10 substitutes as opposed to 100 ). Hence, we consider evaluation metrics that examine the quality and coverage of the top-ranked substitutes from a system with respect to the substitutes that humans judged as acceptable (score > 50 %). Specifically, we compute precision ( P k ) and recall ( R k ) at k 6 : P k = # acceptable substitutes in system topk # substitutes in system topk R k = # acceptable substitutes in system topk min( k, # acceptable substitutes ) Because we care about both quality (precision) and coverage (recall) when comparing systems, we report F k , the harmonic mean of P k and R k . Likewise, we evaluate against the list of substitutes 6 Note that our definition of recall at k is non-standard; the min compensates for the fact that there are often fewer than k acceptable substitutes. which humans judged as conceivable (score > 0 %). P kc and R kc constitute precision and recall of systems against this larger candidate list, and F kc their harmonic mean. Motivated by past work (Mc-Carthy and Navigli, 2007), we primarily examine performance for k = 10 and lemmatize system and reference substitutes during comparison. We note that these metrics represent a departure from standard lexical substitution methodology, established by McCarthy and Navigli (2007). Like P k and R k , the previously-used BEST and OOT metrics are also measures of precision and recall, but do not take advantage of the negative labels from our binary data collection protocol as no such labels existed in the earlier benchmarks. Nevertheless, we report performance of all systems on these metrics in Appendix E as reference. We evaluate both state-of-the-art lexical substitution systems and large-scale pre-trained models as baselines on SWORDS . We reimplement the BERT-based lexical substitution system (BERT-LS) from Zhou et al. (2019), which achieves state-of-the-art results on past benchmarks. As another lexical substitution system, we examine WORDTUNE (AI21, 2020), a commercial system which offers lexical substitution capabilities. 7 We also examine two large-scale pre-trained models adapted to the task of lexical substitution: BERT (Devlin et al., 2019) and GPT -3 (Brown et al., 2020). To generate and rank candidates with BERT , we feed in the context with target word either masked (BERT-M) or kept intact (BERT-K), and output the top 50 most likely words according to the masked language modeling head. Because the target word is removed, BERT-M is expected to perform poorlyits main purpose is to assess the relative importance of the presence of the target word compared to the context. Note that both of these strategies for using BERT to generate candidates differ from that of BERT-LS, which applies dropout to the target word embedding to partially obscure it. To generate candidates with GPT-3, we formulate lexical substitution as natural language generation (see Appendix D.5 for details). lexical substitution systems evaluated on SWORDS . We evaluate the performance of HUMANS using labels from a separate pool of annotators as described in Section 4.4. Because this task is inherently subjective, this system represents the agreement of two independent sets of humans on this task, which should be thought of as the realistic upper bound for all metrics. We consider the substitutes that have score > 0% from the separate pool of annotators as HUMANS 's substitutes in the generative setting. We also consider both of the candidate sources, COINCO and THESAURUS , as oracle systems. Each source contains a list of substitutes for every target word, and therefore can be viewed as a lexical substitution system and evaluated on SWORDS . COINCO provides substitutes for a target word that were provided by (six) human annotators. This can be thought of as a proxy for how humans perform on lexical substitution when recalling words off the top of their heads (as opposed to making binary judgements as in HUMANS ). THESAURUS provides context-free substitutes for a target word (regard-less of their word senses) with the default ranking retrieved from the thesaurus. This represents the context-insensitive ordering that a user of the same thesaurus would encounter. Because these oracle systems only produce candidates which are guaranteed to be in SWORDS , they have an inherent advantage on the evaluation metrics over other systems. Hence, to be more equitable to other systems, we additionally compute F 10 and F 10 c in a lenient fashionfiltering out model generated substitutes which are not in SWORDS (we refer to the setup without filtering as strict). It is our intention that future systems should not use COINCO or THESAURUS in any way, as they leak information about the SWORDS benchmark. Table 4 shows that the performance of all methods falls short of that of humans on all metrics. We interpret this as evidence that SWORDS is a challenging benchmark, since strong (albeit unsupervised) baselines like BERT and GPT -3 do not reach parity with humans. We also observe that two models (WORDTUNE and GPT -3) achieve higher F 10 than COINCO . In other words, while all models perform worse than humans who are judging the appropriateness of substitutes (HUMANS ), some models appear to slightly outperform humans who Lenient Strict Model F 10 F 10 c F 10 F 10 c HUMANS * 51 . are thinking of substitutes off the top of their head (COINCO ). This implies that some lexical substitution models may already be helpful to humans for writing assistance, with room for improvement. Overall, we find that there is no single system which emerges as the best on all metrics. We note that, despite BERT-LS representing the state-of-the-art for past lexical substitution benchmarks, its performance falls short of that of commercial systems like GPT -3 and WORDTUNE on most criteria. Also, the BERT -based methods output around 5 x as many candidates as the other models on average, thus having an inherent advantage in recall with the lenient criteria (see Table 7 in Appendix E). In Table 4, we additionally report the performance of generative models by re-ranking their lists of substitutes using the best ranker from our candidate ranking evaluation, BERT (see Appendix D for details). This procedure unilaterally improves performance for all systems on all metrics except for GPT -3. 8 Hence, we speculate that improved performance on the ranking setting will be complementary to improved performance on the generative setting. From a qualitative perspective, many of the systems we evaluate already produce helpful substi-8 We speculate that this is because GPT-3 produces many substitutes containing multiple word pieces, and mean-pooling several word pieces may result in lower-quality scores. tutes (Table 5). In examining errors, we find that BERT -based models and WORDTUNE tend to produce words that differ semantically from the target (e.g. league for zone). Substitutes generated by GPT -3 are often repetitive (e.g. for zone, GPT -3 produced 64 substitutes, out of which only 13 were unique)we filter out duplicates before evaluating. Finally, we observe that some systems produce appropriate substitutes which are not present in SWORDS (e.g. GPT -3 produces precinct for zone), indicating that SWORDS still has gaps in coverage. However, the higher coverage and quality in SWORDS compared to past benchmarks still improves the reliability of our proposed evaluation. As we already discussed previous lexical substitution benchmarks in Section 2 and models in Section 5, we use this section to draw connections to other related literature. Word sense disambiguation. The task of word sense disambiguation consists of selecting the intended meaning (i.e. sense) from the pre-defined set of senses for that word in a sense inventory. The task of lexical substitution is closely related to word sense disambiguation, as many words are sense synonyms some of their senses are synonymous, but others are not (Murphy, 2010). In fact, McCarthy (2002) proposed lexical substitution as an application-oriented word sense disambiguation task that avoids some of the drawbacks of standard word sense disambiguation, such as biases created by the choice of sense inventory (Kilgarriff, 1997). Near-synonym lexical choice. Words are often near-synonyms they can substitute for each other in some contexts, but not every context (DiMarco et al., 1993; Murphy, 2010). SWORDS can be viewed as a collection of human judgments on when certain near-synonyms are substitutable in a given context. The task of near-synonym lexical choice consists of selecting the original target word from a set of candidate words given a context where the target word is masked out (Edmonds and Hirst, 2002). The candidate words are composed of the target word and its near-synonyms which are often retrieved from a lexical resource such as Hayakawa (1994). In this task, systems are tested whether they can reason about near-synonyms and choose the best substitute that fits in the context, without knowing any direct semantic information about the Context The e-commerce free zone is situated in north Dubai, near the industrial free zone in Hebel Ali Substitutes in SWORDS sector (90%), district (90%), area (90%), region (70%), section (70%), range (60%), strip (60%), ground (50%), segment (50%), territory (50%), sphere (40%), realm (40%), place (30%), tract (30%), city (30%), belt (20%), circuit (20%), band (0%) Reference for F k ( 7 ) sector , district , area , region , section , range , strip Reference for F kc ( 17 ) sector , district , area , region , section , range , strip , ground, segment, territory COINCO ( 9 ) area , region , district , section , city, place, range , strip , territory THESAURUS  ( 14 ) district , area , belt, territory, region , realm, sector , section , circuit, segment WORDTUNE  ( 11 ) district , area , city, region , site, league, center, system, place, zona GPT -3  ( 13 ) district , area , territory, region , realm, sector , locality, section , quarter, precinct BERT-LS ( 50 ) belt, district , port, area , zones, city, park, center, strip , sector BERT-K  ( 50 ) zones, district , area , city, belt, region , park, ville, site, sector BERT-M  ( 50 ) zones, district , area , city, belt, territory, region , haven, park, site Table 5: Qualitative comparison of top 10 candidates generated by best systems. Lexical and phrasal resources. Lexical resources such as thesauri are often used to identify possible word substitutes. WordNet (Fellbaum, 1998) is a widely used lexical resource for English that includes synonymy, antonymy, hypernymy, and other relations between words. PPDB (Pavlick et al., 2015) includes both word-level and phrase-level paraphrase rules ranked by paraphrase quality. These resources relate words and phrases in the absence of context, whereas lexical substitution requires suggesting appropriate words in context. Paraphrase generation. Work on sentence-level paraphrase generation considers a wide range of meaning-preserving sentence transformations, including phrase-level substitutions and large syntactic changes (Madnani and Dorr, 2010; Wieting and Gimpel, 2018; Iyyer et al., 2018; Hu et al., 2019). Our work could be extended to phrases given appropriate methods for identifying target phrases and proposing candidate substitute phrases. One benefit of focusing on word substitutions is that we can cover a large fraction of all appropriate substitutes, and thus estimate recall of generative systems. Some word-level substitutions, such as function word variation and substitutions that rely on external knowledge, are also outside the scope of our work but occur in standard paraphrase datasets (Bhagat and Hovy, 2013). Self-supervised pre-trained models. The task of suggesting words given surrounding context bears strong resemblance to masked language modeling, which is commonly used for pretraining (De-vlin et al., 2019). However, for lexical substitution, appropriate substitutes must not only fit in context but also preserve the meaning of the target word; thus, additional work is required to make BERT perform lexical substitution (Zhou et al., 2019; Arefyev et al., 2020). Modeling human disagreement. In SWORDS , we find considerable subjectivity between annotators on the appropriateness of substitutes. For the task of natural language inference, recent work argues that inherent disagreement between human annotators captures important uncertainty in human language processing that current NLP systems model poorly (Pavlick and Kwiatkowski, 2019; Nie et al., 2020). We hope that the fine-grained scores in SWORDS encourage the development of systems that more accurately capture the graded nature of lexical substitution. We sincerely thank Frieda Rong, Nelson Liu, Stephen Mussmann, Kyle Mahowald, Daniel Jiang, and all reviewers for their help and feedback throughout this project. We also thank OpenAI and Wordtune for allowing us to evaluate their systems. This work was funded by DARPA CwC under ARO prime contract no. W911NF-15-1-0462.
objective abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method result result method method abstain abstain objective abstain method result objective result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain method abstain method abstain method abstain abstain method abstain method abstain abstain method abstain method method method abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method method abstain abstain abstain method method abstain method abstain abstain method method abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain result abstain method other other other other other other other other other other other other other other other other other objective method abstain other other other other abstain other method other other other other