Simile interpretation (SI) and simile generation (SG) are challenging tasks for NLP because models require adequate world knowledge to produce predictions. Previous works have employed many hand-crafted resources to bring knowledge-related into models, which is time-consuming and labor-intensive. In recent years, pre-trained language models (PLMs) based approaches have become the defacto standard in NLP since they learn generic knowledge from a large corpus. The knowledge embedded in PLMs may be useful for SI and SG tasks. Nevertheless, there are few works to explore it. In this paper, we probe simile knowledge from PLMs to solve the SI and SG tasks in the unified framework of simile triple completion for the first time. The backbone of our framework is to construct masked sentences with manual patterns and then predict the candidate words in the masked position. In this framework, we adopt a secondary training process (Adjective-Noun mask Training) with the masked language model (MLM) loss to enhance the prediction diversity of candidate words in the masked position. Moreover, pattern ensemble (PE) and pattern search (PS) are applied to improve the quality of predicted words. Finally, automatic and human evaluations demonstrate the effectiveness of our framework in both SI and SG tasks. The simile, which is a special type of metaphor, is defined as a figurative expression in which two fundamentally different things are explicitly compared, usually using like or as (Israel et al., 2004; Zeng et al., 2020). It is widely used in literature because it can inspire the reader's imagination (Paul, 1970) by giving a vivid and unexpected analogy between two objects with similar attributes. elements: the tenor , the attribute and the vehicle , 1 which can be defined in the form of a triple ( tenor attribute , vehicle ) (Song et al., 2021). For example, the simile sentence Love is as thorny as rose can be extracted as the triple (love, thorny, rose), where the tenor is love, the vehicle is rose, and the attribute is thorny. Note that a simile triple can produce different simile sentences with different templates. For the example triple above, the simile sentences can be also constructed as  love is thorny like rose " with the pattern  tenor is attribute like vehicle ". The study of simile is benefit to many downstream tasks, like sentiment analysis (Rentoumi et al., 2012), question answering (Zheng et al., 2020), writing polishment (Zhang et al., 2021) and creative writing (Gero and Chilton, 2019). Simile interpretation (SI) (Qadir et al., 2016; Su et al., 2016) and simile generation (SG) (Yu and Wan, 2019) are the two important tasks in the study of simile (Tong et al., 2021). The SI task is to find suitable attributes as a mediator between the tenor 1 Tenor: the logical subject of the comparison, usually a noun phrase. Attribute: what things being compared have in common, usually an adjective. Vehicle: the logical object of the comparison, usually a noun phrase. and vehicle. Likewise, the SG task is to select a proper vehicle for the tenor with the given attribution. And these two tasks can be unified into the form of simile triple completion (STC) (Song et al., 2021) as shown in Figure 1. Previous works on the SI and SG tasks relied on a limited training corpus or labor-intensive knowledge base, which leads to an upper limit on the diversity of results. (Song et al., 2021) collected sentences containing comparator words from a Chinese essays corpus and manually annotated them to obtain the simile triple. Some works (Stowe et al., 2021; Gero and Chilton, 2019; Veale et al., 2016) relied on a knowledge base such as ConceptNet 2 , FrameNet 3 , which are scarce to other languages because it is time-consuming and labor-intensive to construct such a knowledge base. It is notable that pre-trained language models (PLMs) (Devlin et al., 2019; Radford et al., 2019) have made significant progress recently in many NLP tasks since it learns generic knowledge such as grammar, common sense from a large corpus (Davison et al., 2019; Liu et al., 2021a,b). Considering the suf-ficient existence of simile in the large corpus, it's reasonable to assume that PLMs are equipped with rich knowledge of similes during the pre-training stage. However, few works have explored directly probing the knowledge of simile from the PLMs. In this paper, we propose a unified framework to solve the SI and SG tasks by mining the knowledge in PLMs, which does not require fine-labeled training data or knowledge graphs. The backbone of our method is to construct masked sentences with manual patterns from an incomplete simile triple, and then use language models with MLM heads to predict the masked words over the task-specific vocabulary. We take the K words with the highest probability as the result words. However, there are problems with this crude approach. Firstly, the predicted words should be creative and surprised for the simile sentence. On the contrary, the PLMs tend to predict common words (e.g., good, bad) with a higher probability. To address this issue, we introduce a secondary pre-training stage Adjective-Noun mask Training (ANT), where only the noun or adjective contained in the amod dependencies (Nivre et al., 2017) could be masked in the MLM training process and the number of words masked times are limited. Secondly, the words predicted 2 https://conceptnet.io/ 3 https://framenet.icsi.berkeley.edu/fndrupal/ by MLM have a preference for different patterns. For this reason, we employ a pattern ensemble to obtain high-quality and robust results. Finally, we also introduce a prompt-search method to improve the quality of the simile component predictions. Our main contributions are as follows:  We propose a unified framework to solve both the simile interpretation (SI) and simile generation (SG) tasks based on pre-trained models. To the best of our knowledge, it is the first work to introduce pre-trained language models to unify these tasks. We propose a secondary pre-training stage that effectively improves the prediction diversity. Further, we employ the pattern-ensemble and pattern-search approaches to obtain better results. We compare our models on both automated metrics and manual measures, and the results show that our approach outperforms the baselines in terms of diversity and correctness. Simile interpretation and simile generation are the two main directions of the simile study (Yu and Wan, 2019). The SI task (Shutova, 2010; Su et al., 2017) aims at finding a suitable attribute when given the tenor and vehicle, while the SG task (Yu and Wan, 2019) is to find a proper vehicle when given the tenor and its attribute. For simile interpretation, some works (Zheng et al., 2020; Bar et al., 2018; Xiao et al., 2016; Gagliano et al., 2016; Qadir et al., 2016) applied word vectors to decide which attribute words can fit into the tenor and vehicle domains and some other works (Gero and Chilton, 2019; Stowe et al., 2021) introduced knowledge base (Baker et al., 1998; Speer et al., 2017) to help find intermediate attributes. For simile generation, some works focused on constructing limited training corpus to finetune a sequence-to-sequence model (Lewis et al., 2020) by pattern-based (Zhang et al., 2021; Bollegala and Shutova, 2013) or knowledge-based approaches (Chakrabarty et al., 2020, 2021; Stowe et al., 2021). There are also some works (Abe et al., 2006; Hervs et al., 2007; Zheng et al., 2020) that focused more on the relationships between concepts (i.e., 5876 BookCorpus Incomplete triple Optimal multi-pattern combination Pattern Search Masked LM LM_ANT Adjective-Noun Mask Training Training Dataset  Beautiful Romantic   Task Vocab T V A The T is as A as V. ... 4 Classes Multi-Patterns T AV A T V ... ... ... tenor and vehicle) and attribute. However, our paper carries out the task of simile interpretation and generation uniformly in the form of simile triples. And instead of extracting the simile triples from the limited corpus using designed templates or a handcrafted knowledge base, we probe simile-related knowledge from PLMs. Pre-trained language models such as Bert and GPT (Devlin et al., 2019; Radford et al., 2019) are trained on the large-scale unlabeled corpus. Many recent works (Manning et al., 2020; Ettinger, 2020; Petroni et al., 2019; Shin et al., 2020; Haviv et al., 2021; Jiang et al., 2020; Zhong et al., 2021; Wang et al., 2022a,b; Li and Liang, 2021) focused on exploring the rich knowledge embedded in these PLMs. Manning et al. (2020) and Ettinger (2020) learned the syntactic and semantic knowledge from PLMs. Among these works, one branch of works(Petroni et al., 2019; Shin et al., 2020; Haviv et al., 2021; Jiang et al., 2020) designed discrete patterns to explore the common sense and world knowledge embedded in PLMs. In addition, some works (Zhong et al., 2021; Li and Liang, 2021) probed knowledge by searching the best-performing continuous patterns in the space of embedding. Inspired by the above works, in this paper, we probe the knowledge of simile in these pre-trained models and further apply pattern ensemble and pattern search to improve the results. As shown in Figure 1, the simile triple complete consists of two tasks: simile interpretation (SI) and simile generation (SG). Each simile sentence can be abstracted into the form of a triple. Therefore, we define a triple: ( T , A , V ) , where T , V are mainly nouns or noun phrases and represent the tenor and vehicle in the simile sentence, respectively. A is the attribute in simile sentences, which is an adjective. If the A is None in the triple, i.e. ( T , None, V ) , we define it as the simile interpretation task. Similarly, if the V is None, i.e. ( T , A , None ) , this will be the task of simile generation. The masked language model (MLM) (Devlin et al., 2019; Taylor, 1953) randomly masks the words in the input sentence and feeds the masked sentence into the pre-trained models to make predictions by other visible words. For example, given a sentence s = [ w 1 , w 2 , . . . , w i , . . . , w m ] , where the w i means the i -th word in the sentence. We can randomly mask s and feed the masked sequence (cid:101) s into the PLMs e.g. BERT (Devlin et al., 2019) to obtain the masked words by Equation: (cid:101) s = f mask ( s, i, v ) (1) P = f  ( (cid:101) s ) (2) 5877 where the v means the Vocabulary for pre-trained models, and the i denotes the position of the masked word in Equation 1. The  is the parameters of PLMs in Equation 2. We can select the word corresponding to the maximum probability in P as the output of the model. To probe the simile knowledge in pre-trained masked language models, the intuitive solution is: (1) Construct a sentence that contains the simile triple in Section 3.1 with the given pattern. (2) Mask the attribute A or vehicle V in this simile sentence. (3) Predict the words in the masked position with MLM. For example, when given a pattern The T is as A as V , the input sentence of MLM is The T is as [MASK] as V for the SI task while The T is as V as [MASK] for the SG task. To formulate this problem, we define the pattern function as p (  ) , where   { SG, SI } . The pre-trained MLM is denoted as M and the predicted distribution Q over vocabulary V can be formulated as: Q ( w | p (  )) = exp ( M ( w | p (  ))) (cid:80) w (cid:48)  V exp ( M ( w (cid:48) | p (  ))) (3) 4 Method In this section, we will introduce our proposed method of probing simile knowledge from pre-trained models. Our method first introduces a secondary pre-training stage Adjective-Noun mask Training (ANT) based on pre-trained language models to acquire diverse lexical-specific words. Then two modules of pattern ensemble and pattern search are used to obtain the high-quality predictions. The framework of our method is shown in Figure 2 in detail 4 . For the MLM task, pre-trained models prefer to output high-frequency words as candidate words since the objective of the training is to minimize the cross-entropy loss (Gehrmann et al., 2019). However, the components of simile triples are usually nouns or adjectives and the simile sentences are appealing due to their creativity and unexpectedness. Therefore, to predict more diverse and 4 we released our code at https://github.com/nairoj/Probing-Simile-from-PLM. specific words of simile component, we introduce a secondary pre-training stage Adjective-Noun mask Training (ANT) that fine-tune the pre-trained model with specially designed datasets. First, we utilize trankit (Nguyen et al., 2021) to construct the training set by selecting sentences from BookCorpus (Zhu et al., 2015) that contains amod 5 dependencies (Nivre et al., 2017). Second, we mask a word at the end of amod relation, instead of randomly masking, and all words are masked no more than 5 times. Finally, the pre-trained model is fine-tuned on the constructed dataset with MLM loss. In this way, the pre-trained model will avoid the bias to high-frequency words and have a higher probability of generating diverse and novel words. Since words predicted by MLM have a preference for different patterns and only using one pattern is insufficient, we apply the pattern ensemble to obtain better performance where different types of patterns are designed as shown in Table 1. Specifically, the class I describes the relationship between the three-element T , V and A . However, the similes tend to highlight an obvious attribute between tenor and vehicle (Israel et al., 2004). We further design the class II and class III to find the attribute corresponding to the tenor and vehicle, respectively. Finally, the attributes of simile sentences are sometimes omitted and thus the class IV is designed to deal with this case. Additionally, we also design three patterns for each class to obtain high-quality and robust results. where P is the set of patterns p (  ) for specific task  . Note that though we design four classes of patterns in Table 1, some classes of patterns are not required for the SI or SG task. Specifically, The patterns of Class IV are not used for the SI task because the attribute A is missed in Class IV. Likewise, the patterns of Class III are not used for the SG task due to the lack of vehicle V . 5 An adjectival modifier of a noun (or pronoun) is any adjectival phrase that serves to modify the noun (or pronoun). The relation applies whether the meaning of the noun is modi-fied in a compositional way (e.g., large house) or an idiomatic way (hot dogs). The prediction of pattern ensemble in Section 4.2 is averaged by adding up the output distributions of all the patterns. Conversely, the hand-designed patterns are heuristic, which may lead to suboptimal results. Therefore, it is worth studying how these patterns can be combined to obtain better performance. To solve this problem, we introduce an approach of pattern search (PS) to find the best combination of different patterns. Specifically, given a simile dataset DPS , we calculate Equation 4 on DPS by iterating all subsets of the patterns. Finally, we select the optimal subset p best as the input of MLM to predict simile components. Dataset for ANT: We constructed our train set of ANT from BookCorpus. We first extracted the sentences with length less than 64 and then masked nouns or adjectives in them based on amod dependencies (Nivre et al., 2017). Meanwhile, we limited the frequency of masked words to less than 5. Finally, we got 98k sentences as the dataset of ANT, which contains 68k noun-masked sentences and 30k adjective-masked sentences. Dataset for PE and PS: We evaluate our method on the dataset proposed in (Roncero and de Almeida, 2015). As the samples in Table 2, there are multiple attributes for each ( T , V ) pair. For example, the pair of (anger, fire) has the attributes of dangerous, hot, and red. In addition, we followed the previous work (Xiao et al., 2016) to filter the dataset by reversing simile triples with attribute frequencies greater than 4. Eventually, we obtain the train set with 533 samples and the test set with 145 samples. Notice that the train set is the Triple Frequency (Anger, Dangerous, Fire) 8 (Anger, Hot, Fire) 8 (Anger, Red, Fire) 5 (Love, Beautiful, Rainbow) 10 (Love, Beautiful, Melody) 2 (Love, Beautiful, Rose) 9 Table 2: Some samples from the dataset. DPS in Section 4.3 used for the pattern search and the test set is used for evaluating all the approaches in this paper. Details for ANT : In adjective-nouns mask training, we utilized Adam as our optimizer and the learning rate is 5e-5. The batch size is set to 32 and the max sequence length is set to 64, respectively. Further, we utilize the Bert-Large 6 with 340M parameters as the basic model to perform adjective-nouns mask training and the number of training epoch is 3 . Vehicle Filtering : For simile generation, we filter the predicted vehicles that are similar to the tensor by calculating the semantic similarity with Glove embedding. For instance, given the sentence The child is as tall as [MASK]", we will filter out the word father" as its vehicle due to not meeting the simile definition 7 . To solve this problem, we compute the similarity score of the tenor and vehicle and filter the predicted vehicle whose score is above the threshold 0.48 8 . 6 https://huggingface.co/Bert-large-uncased 7 Using something with similar characteristics to give an analogy to another thing 8 The threshold is the maximum similarity score of tenor and vehicle in the train set 5879 72.4 67.1 48.5 46.1 35.4 25.0 1.3 21.1 29.0 9.2 5.3 1.3 good big strong old real great 0 20 40 60 80 100 P e r c en t age ( % ) BertANT Top 15 84.2 76.3 69.7 67.1 55.3 42.1 5.3 29.0 32.9 17.1 11.8 1.3 good big strong old real great 0 20 40 60 80 100 P e r c en t age ( % ) BertANT Top 25 Figure 3: Percentage of samples whose top K predicted words contain a given common word. In this section, we will demonstrate that ANT could improve the diversity of predicted words for both the SI and SG tasks. We compare the predicted results of MLM (i.e., Bert) before and after ANT, which use the patterns The T is as [MASK] as V " for the SI task and The T is as A as [MASK]" for the SG task. Metric: We evaluate the diversity of the MLM predictions by calculating the proportion of unique words in the predicted Top K results on the test set. It can be formulated as p @ K = Num ( Unique _ words ) K  N (5) where the Num ( Unique _ words ) means the number of unique words, and the N represents size of the test set. Result: To illustrate the effectiveness of ANT, We evaluate the results on the test set based on Equation 5. As shown in Table 3, the diversity of predicted words significantly improves after ANT for different p @ K , specifically about 100% improvement for the SI task and about 50% for the SG task. Additionally, Figure 3 plots the percentage of samples on the test set, where a given common word (e.g., good, big, strong) appears in the list of the top k = 15 , 25 predicted words. We can observe that the frequency of common words decreases significantly after ANT. For example, the frequency of the common word good decreases from 72 . 37% to 1 . 32% when k = 15 . We compare the proposed approaches with the following baseline: (1) Meta4meaning (Xiao et al., 2016) : It uses the trained LSA vector representation according to the degree of abstraction and salience imbalance to select appropriate attributes. (2) GEM (Bar et al., 2018) : A method calculates the cosine similarity and normalized PMI between each attribute and tensor/vehicle based on Glove representing to obtain the best attribute with ranking. (3) Bert (De-vlin et al., 2019) : Directly use pre-trained MLM to predict the simile component with a single pattern as Section 3.3. In this paper, we utilize the bert-large-uncased as the basic pre-trained MLM. (4) ConScore (Zheng et al., 2020) : A connecting score is proposed to select an attribute word A for T and V . Our proposed approaches are denoted as: (1) ANT : Perform Adjective-Noun mask Training based on a pre-trained MLM with the datasets mentioned in Section 5.1. (2) ANT+PE : Based on ANT, the output distribution over vocabulary is pre-5880 Task Method MRR R@5 R@10 R@15 R@25 R@50 Meta4meaning N/A 0.221 0.303 0.339 0.397 0.454 GEM 0.312 0.198 0.254 0.278 0.405 0.562 ConScore 0.078 0.076 0.138 0.172 0.269 0.386 Bert 0.266 0.338 0.428 0.448 0.538 0.641 ANT 0.245 0.310 0.407 0.455 0.510 0.614 ANT+PE 0.241 0.331 0.400 0.448 0.552 0.628 SI ANT+PS+PE 0.270 0.379 0.490 0.524 0.579 0.655 ConScore 0.036 0.055 0.09 0.103 0.145 0.200 Bert 0.064 0.076 0.124 0.159 0.207 0.283 ANT 0.049 0.069 0.117 0.145 0.186 0.303 ANT+PE 0.036 0.034 0.083 0.097 0.131 0.172 SG ANT+PS+PE 0.095 0.124 0.145 0.159 0.214 0.290 Table 4: Automatic evaluation for SI and SG tasks. dicted by average on all the corresponding patterns in Table 1. (3) ANT+PS+PE : Based on ANT, first the pattern search is to decide which patterns in Table 1 are applied, and then the pattern ensemble is used over these selected patterns. Automatic Evaluation: (1) Mean Reciprocal Rank (MRR): average on the reciprocal of the ranking r i of label words in the predicted candidates, denoted as MRR = 1 NN (cid:88) i =1 1 r i (6) (2) R @ K : the percentage of the label words appear in the top K predictions. previous works (Xiao et al., 2016; Bar et al., 2018), we consider a predicted word as the correct answer if it is a synonym of label word n in WordNet (Miller, 1992). It can be formulated as cor ( w ) = (cid:40) 1 w  Synonyms ( L i ) 0 w /  Synonyms ( L i ) (7) R @ K = 1 NN (cid:88) i =1 (cid:80) w  K i cor ( w ) K (8) where K i denotes the list of predicted words, L i denotes the list of label words and Synonyms ( L i ) represents the synonyms of a word. Human Evaluation: To further prove our approaches are better than baselines, human evaluation is used to evaluate the quality of predicted simile triples from three levels (0, 1, 2). 0 The triple is unacceptable. 1 The triple is acceptable. 2 The triple is acceptable and creative. Given a simile triple, annotators need to score it according to their subjective judgment and each triple is annotated by three annotators independently. We use the average score of three annotators as the quality of a simile triple. Automatic and Human Evaluation: The results of both automatic and human evaluation are shown in Table 4 and Table 5. The agreement between annotators is measured using Fleiss's kappa  (Ran-dolph, 2005). The  value is 0.68 (substantial agreement) for the SI task and 0.48 (moderate agreement) for the SG task. (1) For both SI and SG tasks, our proposed approaches (i.e., ANT, ANT+PE, ANT+PS+PE) significantly outperform the baselines on both automatic and human evaluations. It proves that our methods not only enhance the diversity of predicted simile components in Section 5.3 but also their quality. (2) Pre-trained MLM-based methods (i.e., Bert, ANT, ANT+PE and ANT+PS+PE) perform better than the traditional methods (i.e., GEM, Meta4meaning, ConScore). It shows the potential of pre-trained models in probing simile knowledge. (3) Compared ANT with Bert, we found that though ANT improves the diversity of predicted words in Table 3, the average scores on automatic and human evaluations decrease because the simile knowledge is not involved in the ANT training process. However, our proposed PE and PS compensate for the performance. (4) The scores of automatic evaluation metrics on the SI task are remarkably higher than the SG task. Yet, the scores of human evaluation metrics are significantly lower than on the SG task. We conjecture that this may be because the list of candidate words of attribute predicted by SI are smaller than that of the vehicle for the SG task. For example, given the SI sample (Cloud, None , Cotton), the attribute words are almost restricted to the physical properties of the vehicle, such as Soft, while the choices of vehicle words are more varied and unexpected given the SG sample (Cloud, soft, None ) such as cotton, silk, towel". In future work, we will continue to study how 5882 to mine broader or complex knowledge from pre-trained models, such as metaphor, common sense and we expect more researchers to perform related research. Discussion for PS: Compared ANT+PS+PE to ANT+PE, it can be included that pattern search brings a great improvement to the results on both automatic and human evaluations. To have a deeper insight into PS, the pattern subsets with high performance are listed in Table 6. For the SI task, the optimal multi-pattern combination is { p 1 , p 5 } , which support the hypothesis proposed by (Ortony, 1979) considers that the highlighted attribute of a simile triple is more salient in the vehicle domain despite it is commonly shared by both tenor and vehicle domains. Specifically, pattern p 1 belonging to the Class I, models the relationship of all three simile components while the pattern p 5 belonging to Class II requires the candidate words to be the salient attribute of the vehicle. Similarly, for SG task, optimal multi-pattern combination is { p 3 , p 4 } , which is also a combination of the Class I pattern and the Class II pattern. In this paper, from the perspective of simile triple completion, we propose a unified framework to solve the SI and SG tasks by probing the knowledge of the pre-trained masked language model. The backbone of our method is to construct masked sentences with manual patterns from an incomplete simile triple, and then use language models with MLM heads to predict the masked words. Moreover, a secondary pre-training stage (the adjective-noun mask training) is applied to improve the diversity of predicted words. Pattern ensemble (PE) and pattern search (PS) are further used to improve the quality of predicted words. Finally, automatic and human evaluations demonstrate the effectiveness of our framework in both SI and SG tasks. This work is supported by the Key Research and Development Program of Zhejiang Province (No. 2022C01011) and National Natural Science Foundation of China (Project 61075058). We would like to thank the anonymous reviewers for their excellent feedback.
abstain abstain abstain abstain abstain objective method method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain other other other other other other abstain method other other other other other abstain abstain abstain method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain result result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective method abstain abstain objective other other