Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains understudied. We introduce the IMPLI ( I diomatic and M etaphoric P aired L anguage I nference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors. We develop novel methods to generate 24k semiautomatic pairs as well as manually creating 1.8k gold pairs. We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset. We then show that while they can reliably detect entailment relationship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing. This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future improvements in this direction. 1 1 Introduction Understanding figurative language (i.e., that in which the intended meaning of the utterance differs from the literal compositional meaning) is a particularly di  cult area in NLP (Shutova, 2011; Veale et al., 2016), but is essential for proper natural language understanding. We consider here two types of figurative language: idioms and metaphors. Idioms can be viewed as non-compositional multiword expressions (Jochim et al., 2018), and have been historically di  cult for NLP systems. For instance, sentiment systems struggle with multiword expressions in which individual words do not directly contribute to the sentiment (Sag et al., 2002). Metaphors involve linking conceptual properties of two or more domains, and are known to be pervasive in everyday language (Lako  and Johnson, 1980; Stefanowitsch and Gries, 2008; Steen et al., 2010). Recent work has shown that these types of figurative language are impactful across a broad array of NLP tasks (see 2.1). Large-scale pre-training and transformer-based architectures have yielded increasingly powerful language models (Vaswani et al., 2017; Devlin et al., 2019; Liu et al., 2019). However, relatively little work has explored these models' representations of figurative and creative language. NLI datasets have widely been used for evaluating the performance of language models (Dagan et al., 2006; Bowman et al., 2015a; Williams et al., 2018), but there are insu  cient figurative language datasets in which a literal sentence is linked to a corresponding figurative counterpart that are large enough to be suitable for evaluating NLI. Due to the creative nature of human language, creating a dataset of diverse, high-quality literal / figurative pairs is time-consuming and di  cult. To address this gap, we build a new English dataset of paired expressions designed to be leveraged to explore model performance via NLI. Our dataset, IMPLI ( I diomatic / M etaphoric P aired L anguage I nference), is comprised of both silver pairs, which are built using semi-automated 5375 methods (3.1), as well as hand-written gold pairs (3.4), crafted to reflect both entailment and non-entailment scenarios. Each pair consists of a sentence containing a figurative expression (id-ioms / metaphors) and a literal counterpart, designed to be either entailed or non-entailed by the figurative expression (Table 1 shows some examples). Our contribution thus consists of three key parts:  We create a new IMPLI dataset consisting of 24,029 silver and 1,831 gold sentence pairs consisting of idiomatic and metaphoric phrases that result in both entailment and non-entailment relationship (see Table 2). We evaluate language models in an NLI setup, showing that metaphoric language is surprisingly easy, while non-entailing idiomatic relationships remain extremely di  cult. We evaluate model performance in a number of experiments, showing that incorporating idiomatic expressions into the training data is less helpful than expected, and that idioms that can occur more in more flexible syntactic contexts tend to be easier to classify. Figurative language includes idioms, metaphors, metonymy, hyperbole, and more. Critically, figurative language is that in which speaker meaning (what the speaker intends to accomplish through an utterance) di  ers from the literal meaning of that utterance. This leads to problems in NLP systems if they are trained mostly on literal data, as their representations for particular words and / or phrases will not reflect their figurative intended meanings. Figurative language has a significant impact on many NLP tasks. Metaphoric understanding has been shown to be necessary for proper machine translation (Mao et al., 2018; Mohammad et al., 2016). Sentiment analysis also relies critically on figurative language: irony and sarcasm can reverse the polarity of a sentence, while metaphors and idioms may make more subtle changes in the speaker meaning (Ghosh et al., 2015). Political discourse tasks including bias, misinformation, and political framing detection benefit from joint learning with metaphoricity (Huguet Cabot et al., 2020). Figurative language engendered by creativity on social media also poses di  culty for many NLP tasks including identifying depression symptoms (Yadav et al., 2020; Iyer et al., 2019) and hate speech detection (Lemmens et al., 2021). We are here focused on idioms and metaphors. There is currently a gap in diagnostic datasets for idioms, and our work fills this gap. There exist some relevant metaphoric resources (see  2 . 2); metaphors are known to be extremely common and important to understanding figurative language, our resource serves to build upon this work. Natural language inference is the task of predicting, given two fragments of text, whether the meaning of one ( premise ) entails the other ( hypothesis ) (Da-gan et al., 2006). The task is formulated as a 3-way classification problem, in which the premise and hypothesis pairs are labeled as entailment , contradiction , or neutral , if their relationship could not be directly inferred (Bowman et al., 2015b). NLI has been widely used as an evaluation task for language understanding, and there have been a large number of challenging datasets, which have been used to further our understanding of the capabilities of language models (Wang et al., 2018, 2019). Paired data for figurative language is relatively sparse, and there is a gap in the diagnostic datasets used for NLI in this area. Previous work includes the literal / metaphoric paraphrases of Mohammad et al. (2016) and Bizzoni and Lappin (2018), although both contain only hundreds of samples, insu  cient for proper model training and evaluation. With regard to NLI, early work proposed the task of textual entailment as a way of understanding metaphor processing capabilities (Agerri et al., 2008; Agerri, 2008). Poliak et al. (2018) build a dataset for diverse NLI, which includes some creative language such as puns, albeit making no claims with regard to figurativeness. Zhou et al. (2021) build a dataset consisting of paired idiomatic and literal expressions. They be-gin with a set of 823 idiomatic expressions yielding 5,170 sentences, and had annotators manually rewrite sentences containing these idioms as literal expressions. We expand on this methodology by having annotators only correct definitions for the idioms themselves and use these definitions to automatically generate the literal interpretations of the idioms by replacing them into appropriate contexts: this allows us to scale up to over 24k silver sentences. We also expand beyond paraphrasing by incorporating both entailment and non-entailment 5376 Fig. Type Ent. Similar to this work, Chakrabarty et al. (2021a) build a dataset for NLI based on figurative language. Their dataset consists of figurative / literal pairs recast from previously developed simile and metaphor datasets, along with a parallel dataset between ironic and non-ironic rephrasing. This sets the groundwork for figurative NLI, but the dataset is relatively small outside of the irony do-main, and the non-entailments are generated purely by replacing words with their antonyms, restricting the novelty of the hypotheses. Their dataset is relatively easy for NLI models; here we show that figurative language can be challenging, particularly with regard to non-entailments. Zhou et al. (2021) and Chakrabarty et al. (2021a) provide invaluable resources for figurative NLI; our works aims to covers gaps in a number of areas. First, we generate a large number of both entailment and non-entailment pairs, allowing for better evaluation of adversarial non-entailing examples. Second, our silver methods allow for rapid development of larger scale data, allowing for model training and evaluation. We show that while entailment pairs are relatively easy (accuracy scores ranging from .86 to .89), the non-entailment pairs are exceedingly challenging, with the roberta-large model achieving accuracy scores ranging from .311 to .539. Our IMPLI dataset is built from idiomatic and metaphoric sentences paired with entailing and non-entailing counterparts, from both silver pairs (3.1) and manually written sentences (3.4). For our purposes, we follow McCoy et al. (2019) in conflating the neutral and contradiction categories into a non-entailment label. We then label every pair as either entailment (  ) or non-entailment ( (cid:57) ). Due to the di  cult nature of the task and to avoid issues with crowdsourcing (Bowman et al., 2020), we employed expert annotators. We used two fluent English speakers, both graduate students in linguistics with strong knowledge in figurative language, paid at a rate of $20 / hr. For each method below, we ran pilot studies, incorporated annotator feedback and iteratively assessed the viability of identifying and generating appropriate expressions. As the annotators were working on generating new expressions, agreement was not calculated: we instead assessed the quality of the resulting expressions (see Section 3.3). Table 2 contains an overview of the di  erent entailment and non-entailment types collected (Detail examples are also provided in Appendix D). First, we explore a method for generating silver pairs using annotators to create phrase definitions which can be inserted automatically into relevant contexts, yielding a large number of possible entailment and non-entailment pairs that di  er only with regard to the relevant phrase. Our procedure hinges on a key assumption: for any given figurative phrase, we can generate a contextually indepen-dent literal paraphrase. We then replace the original expression with the literal paraphrase, following the assumption that the figurative expression necessarily entails its literal paraphrase: He's stuck in bed, which is his hard cheese . He's stuck in bed, which is his bad luck . Conversely, in contexts where the original phrase is used literally, replacing it with the literal paraphrase should yield a non-entailment relation. To build idiomatic pairs, we use three corpora that contain sentences with idiomatic expressions (IEs) labelled as either figurative or literal. 2 These are the MAGPIE Corpus (Haagsma et al., 2020), the PIE Corpus (Adewumi et al., 2021), and the SemEval 2013 Task 5 (Korkontzelos et al., 2013). We collect the total set of IEs that are present in these corpora. We then extract definitions for these using freely available online idiom dictionaries. 3 These definitions are often faulty, incomplete, or improperly formatted. We employed annotators to make manual corrections. The annotators were given the original IE as well as the definition extracted from the dictionary. The annotators were asked to ensure that the dictionary definition given was (1) a correct literal interpretation and (2) fit syntactically in the same environments as the original IE. If the definition met both of these criteria, the IE can be replaced by its definition to yield an entailment pair. If either criterion was not met, annotators were asked to minimally update the definition so that it satisfied the requirements. In total this process yielded 697 IE definitions. We then used the above corpora, replacing these definitions into the original sentences (see Figure 1). We use the figurative / literal labels from the 2 We here use "idiomatic expression" or "IE" to refer to the specific idiom in question (ie. "kick the bucket", "spill the beans"), as opposed to the sentence / context containing it. original corpora: replacing them into figurative contexts yields entailment relations, while replacing them into contexts where the phrase is meant literally then yields non-entailments. As a second method for generating non-entailment pairs, we asked annotators to write novel, adversarial definitions for IEs. Given a particular phrase, they were instructed to invent a new meaning for the IE that was not entailed by the true meaning, but which seemed reasonable presuming they had never heard the original IE. Some examples of this process are shown in Table 3. We then replace these adversarial definitions into figurative sentences from the corpora. This yields pairs where the premise is an idiom used figura-tively, and the hypothesis is a sentence that attempts to rephrase the idiom literally, but does so incorrectly, thus yielding non-entailments (Figure 2). Metaphors are handled in a similar way: we start with a collection of minimal metaphoric expressions (MEs). These are subject-verb-object and adjective-noun constructions from Tsvetkov et al. (2014). Each is annotated as being either literal or metaphoric, along with an example sentence. We passed these MEs directly to annotators, who were then instructed to replace a word in the ME so that it would be considered literal in a neutral context. These can then be replaced in a similar fashion: we start with the original figurative sentence, replace the ME with the literal replacements, and the result is an entailing pair with the metaphoric sentence entailing the literal. We apply this procedure to the dataset of Tsvetkov et al. (2014), yielding 100 metaphoric / literal NLI entailment pairs. We then take a portion of the Common Crawl dataset 4 , and identify sentences that contain these original MEs. We identify sentences that contain the words from the metaphoric phrase, and replace the metaphoric word itself with its literal counterpart. This yields 645 additional silver pairs. For all silver methods, we also employ syntactic postprocessing to overcome a number of hurdles. First, phrases used idiomatically often follow different syntactic patterns than when used literally. Original: These point out of this world , but where to is not made clear. Replaced: *These point wonderful , but where to is not made clear. This phrase in literal contexts functions syntactically as a prepositional phrase, while idiomatically it is used as an adjective. When replaced with the definition "wonderful" in a literal context, we get a grammatically incoherent sentence. Second, phrases in their literal usage often do not form full constituents, due to the string-matching approach of the original datasets. Many literal usages of 4 https://commoncrawl.org/ these phrases are thus incompatible with the defined replacement. I think [this one has to die ] for the other one to live. Turn in [ the raw edges] of both seam allowances towards each other and match the folded edges. To avoid these issues, we ran syntactic parsing on the definition and the expression within each context, requiring that the expression in context begins with the same part of speech as the definition and that it does not end inside of another phrase. Additionally, for each replacement, we ensured that the verb conjugation matched the context. For this, we identified the conjugation in the context, and used a de-lemmatization script to conjugate the replacement verb to match the original. In implementing and analyzing this procedure, we noted a number of practical issues. First, a large number of the MEs provided are actually idiomatic or proverbial: the focus word does not actually contribute to the metaphor, but rather the entire expression is necessary. Similarly, we found that replacing individual parts of MEs is often insu  cient to fully remove the metaphoric meaning. We iterated over possible solutions to circumvent these issues and found that it is best to simply skip instances for which a replacement does not yield a feasible literal interpretation. In order for these automatically created pairs to be useful for NLI-based evaluation, they need to be of su  ciently high quality. As the annotators were generating novel definitions and pairs, rather than inter-annotator agreement, we instead evaluate the quality of the resulting pairs by testing whether the automatically generated pairs contained the appropriate entailment relation. For this task, each annotator was given 100 samples for each general category of silver generations (idiomatic entailments, idiomatic non-entailments, and metaphoric entail-ments). They were asked if the entailment relation between the two sentences was as expected. An expert than adjudicated disagreements to determine the final percentage of valid pairs. To evaluate the syntactic validity of the generated pairs, we additionally ran the Stanford PCFG dependency parser (Klein and Manning, 2003) on 5379  Idioms (cid:57) Idioms  Met. the pairs. Per previous work in NLI (Williams et al., 2018), we evaluate the proportion of sentences for which the root node is S. Table 4 shows the results. The semi-supervised examples evoked the correct entailment relation between %88 and %97 of the time: while there is still noise present, this indicates the e  ectiveness of the proposed methods. With regard to syntax, we see S node roots for between 82% and %90 of the sentences: within the range of the SNLI performance (74%-88%), and slightly behind the MNLI (91%-98%). We find that the generated hypotheses are not significantly di  erent in quality than the premises. This indicates that the method for generation preserves the original syntax. These methods allow us to quickly generate a substantial number of high-quality pairs to evaluate NLI systems on figurative language. However, they may introduce additional bias as we employ a number of restrictions in order to ensure syntactic and semantic compatibility, and we lack full non-entailment pairs for metaphoric data. We therefore expand our dataset with manually generated pairs. To create gold pairs, annotators were given a figurative sentence along with the focus of the figurative expression: for idioms, this is the IE; for metaphors, the focus word of the metaphor. For idioms, we used the MAGPIE dataset to collect contextually figurative expressions. For metaphors, we collected metaphoric sentences from the VUA Metaphor Corpus (Steen et al., 2010), the metaphor dataset of (Mohammad et al., 2016), and instances from the Gutenberg poetry corpus (Jacobs, 2018) annotated for metaphoricity (Chakrabarty et al., 2021b; Stowe et al., 2021) . Annotators were instructed to rewrite the sentence literally. This was done by removing or rephrasing the figurative component of the sentence. This yields gold standard paraphrases for idiomatic and metaphoric contexts. encouraged to keep as much of the original utterance as possible, ensuring high lexical overlap, while removing the main figurative element of the sentence. For idioms, this comes from adding or adjusting words to force a literal reading of the idiom:  The old girl finally kicked the bucket. For metaphors, this typically involves keeping the same phrasing while adapting the sentence to have a di  erent, non-metaphoric meaning. Previous work in NLI has employed the technique of replacing words in the literal sentences with their antonyms to yield non-entailing pairs (Chakrabarty et al., 2021a). We replicate this process for idioms: for the manually elicited definitions, we replace key words as determined by annotators with their antonyms. This yields sentences which negate the original figurative meaning and are thus suitable non-entailment pairs. Previous work found this antonym replacement for figurative language remains relatively easy for NLI systems, which we can additionally explore with regard to idioms. These manual annotations provide a number of concrete benefits. First, they are not restricted to individual words or phrases (excluding antonyms): the figurative components can be rewritten freely, allowing for diverse, interesting pairs. Second, they are written by experts, ensuring higher quality than the automatic annotations, which may be noisy. Using the IMPLI dataset, we aim to answer a series of questions via NLI pertaining to language models' ability to understand and represent figurative language accurately. These questions are:  R1: How well do pre-trained models perform on figurative entailments and non-entailments? R2: Does adding idiomatic pairs into the training data a  ect model performance? R3: Does the flexibility of idiomatic expressions a  ect model performance? in previous work: it contains a large number of both entailments and non-entailments and is large enough to be used for training the models. We obtain baseline NLI models by fine-tuning roberta-base and roberta-large models on the MNLI dataset (Williams et al., 2018), with entailments as the positive class and all others as the negative and evaluate them on their original test sets as well as IMPLI . 5 Due to variance in neural model performance (Reimers and Gurevych, 2017), we take the mean score over 5 runs using di  erent seeds. We report results in Table 5. We observe that idiomatic entailments are relatively easy to classify, with accuracy scores over .84. Non-entailments were much more challenging. Silver pairs generated through adversarial definitions were especially di  cult: the pairs contain high lexical overlap, and in many cases the premise and hypotheses are semantically similar. The replacement into literal samples were easier, as the idiomatic definition clashes more starkly with the original premise, making non-entailment predictions more likely. Consistent with Chakrabarty et al. (2021a)'s work in metaphors, non-entailment through antonym replacement is easiest for idioms: the antonymic relationship can be a marker for non-entailment, despite the high word overlap. With regard to metaphors, silver entailment pairs are relatively easy. Manual pairs are more challenging but are still much easier than idioms. This is supported by the fact that metaphors are common in everyday language: these models have likely seen the same (or similar) metaphors in training. Our findings show that in fact metaphoricity may not be particularly challenging for deep pre-trained models, as they are able to e  ec-tively capture the metaphoric entailment relations. The roberta-large model performs better for metaphoric expressions than roberta-base , but 5 Model hyperparameters found in Appendix A. the di  erence on other partitions is relatively small. 6 We also find that lexical overlap plays a significant role here as noted by previous work (McCoy et al., 2019): sentences with high overlap tend to be classified as entailments regardless of the true label (for more, see Appendix B). We note that the manual pairs tend to be more di  cult for both idioms and metaphors: these pairs can be more flexible and creative, whereas the silver pairs are restricted to more regular patterns. R2: Incorporating Idioms into Training To evaluate incorporating idioms into training, we then split the idiom data by idiomatic phrase types, keeping a set of IEs separate as test data to assess whether the model can learn to correctly handle novel, unseen phrases. Our goal is to assess whether poor performance is due to models' not containing these expressions in training, or because their ability to represent figurative language inherently limited. We hypothesize that the non-compositional nature of these types of figuration should lead to poor performance on unseen phrases, even if the model is trained on other idiomatic data. For each task, we split the data into 10 folds by IE and incrementally incorporate these folds into the original MNLI for training, leaving one fold out for testing. We experiment with incorporating all training data for both labels, as well as using only entailment or non-entailment samples. We then evaluate our results on the entire test set, as well as the entailment and non-entailment partitions. Figure 4 shows the results, highlighting that additional training data yields only small improvements. Pairs with non-entailment relations remain exceedingly di  cult, with performance capping out at only slightly better than chance. As hypothesized, additional training data is only somewhat e  ective in improving language models' idiomatic capabilities; this is not su  cient to overcome di  culties from literal usages of idiomatic phrases and adversarial definitions, indicating that idiomatic 6 We found minimal di  erences between these models across R1-R3. R3: Syntactic Flexibility Finally, we assess models' representation of idiomatic composition-ality. Nunberg et al. (1994) indicate that there are two general types of idioms: "idiomatic phrases", which exhibit limited flexibility and generally occur only in a single surface form, and "idiomatically combining expressions" or ICEs, in which the constituent elements of the idiom carry semantic meaning which can influence their syntactic properties, allowing them to be more syntactically flexible. For example, in the idiom spill the beans , we can map the spilling activity to divulging of information, and the beans to the information. Because this expression has semantic mappings to figurative meaning for its syntactic constituents, Nunberg et al. (1994) argue that it can be more syntactically flexible, allowing for expressions like the beans that were spilled by Martha to maintain idiomatic meaning. For fixed expressions such as kick the bucket , no syntactic constituents map directly to the figurative meaning ("die"). We then expect less syntactic flexibility, and thus the bucket that was kicked by John loses its idiomatic meaning. We hypothesize that model performance will be correlated with the degree to which a given idiom type is flexible: more fixed expressions may be easier, as they are seen in regular, fixed patterns that the models can memorize, while more flexible ICEs will be more di  cult, as they can appear in di  erent patterns, cases, and word order, often even mixing in with other constituents. To test this, we define an ICE score as the percentage of times a phrase occurs in our test data in a form that does not match its original base form. Higher percentages mean the phrase occurs more frequently in a non-standard form, acting as a measure for the syntactic flexibility of the expression. We assessed the performance of the roberta-base model for each idiom type, evaluating Spearman correlations between performance and idioms' ICE scores. We found no correlation between ICE scores and performance for entailments, nor for adversarial definition non-entailments ( r = . 004 /. 45, p = . 921 /. 399, see Appendix C). However, we do see a weak but significant correlation ( r = . 188, p = 0 . 016) with non-entailments from literal contexts: the model performs better when the phrases are more flexible, contrary to our initial hypothesis. One possible explanation is that the model memorizes a specific figurative meanings for each fixed expression, disregarding the possibility of these words being used literally. When the expression is used in a literal context, the model then still assumes the figurative meaning, resulting in errors on non-entailment samples. The ICEs are more fluid, and thus the model is less likely to have a concrete representation for the given phrase: it is better able to reason about the context and interacting words within the expression, making it easier to distinguish the entailing and non-entailing samples. In this work, we introduce the IMPLI dataset, which we then use to evaluate NLI models' capabilities on figurative language. We show that while widely used MNLI models handle entailment admirably and metaphoric expressions are relatively easy, non-entailment idiomatic relationships are more di  cult. Additionally, adding idiom-specific training data fails to alleviate poor performance for non-entailing pairs. This highlights how currently language models are inherently limited in representing some figurative phenomena and can provide a target for future model improvements. For future work, we aim to expand our data collection processes to new data sources. Our dataset creation procedure relies on annotated samples and definitions: as more idiomatic and metaphoric resources become available, this process is broadly extendable to create new figurative / literal pairs. Additionally, we only explore this data for evaluating NLI systems: this data could also be used for other parallel data tasks such as figurative language interpretation (Shutova, 2013; Su et al., 2017) and figurative paraphrase generation. As natural language generation often relies on training or fine-tuning 5382 models with paired sentences, this data could be a valuable resource for figurative language generation systems.
abstain abstain abstain objective method result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain objective result result other other other other other other other other method abstain abstain other other method other other other other other other method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain method method abstain abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method method method abstain method abstain abstain abstain abstain method abstain other abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method result abstain abstain objective objective abstain abstain