While large language models have shown exciting progress on several NLP benchmarks, evaluating their ability for complex analogical reasoning remains under-explored. Here, we introduce a high-quality crowdsourced dataset of narratives for employing proverbs in context as a benchmark for abstract language understanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs, ensuring that models need to go beyond surface-level reasoning to succeed. We explore three tasks: (1) proverb recommendation and alignment prediction, (2) narrative generation for a given proverb and topic, and (3) identifying narratives with similar motifs. Our experiments show that neural language models struggle on these tasks compared to humans, and these tasks pose multiple learning challenges. Large language models (LLMs) (Devlin et al., 2019; Liu et al., 2019a; Raffel et al., 2020; Lewis et al., 2020; Reimers and Gurevych, 2019; Sanh et al., 2019; Lan et al., 2020) have led to a paradigm shift in NLP, and have shown exciting progress on benchmarks such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a). In particular, these include tasks such as reading comprehension, natural language inference, and coreference resolution. Many of these tasks rely on semantic and syntactic reasoning, which has been mastered by these LLMs. For example, apart from improving on distributional semantics through contextual-ized embeddings (Ethayarajh, 2019), recent work has shown evidence that these models implicitly learn emergent concepts such as subject-verb agreement (Jawahar et al., 2019), semantic roles (Tenney et al., 2019) and dependency structures (Hewitt and Manning, 2019). However, humans show an ability for deeper linguistic reasoning. We can identify people's intentions and goals (Douglas and Sutton, 2006), perform relational reasoning (Alexander et al., 2016), and find analogies in situations with little surface overlap (Holyoak, 2013). In particular, making verbal analogies in the form of proverbs is noted as an indicator of literary ability (Penfield and Duru, 1988; Nippold et al., 2001). Proverbs are also repositories of information on culture, societal norms, values, and folk wisdom (Raymond, 1956; White, 1987). In this work, we investigate proverbs in narrative contexts as a testbed for evaluating abstract reasoning and analogical abilities of LLMs. Context), a high-quality crowdsourced dataset of narratives paired with proverbs. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and is designed to minimize lexical overlap between narratives and proverbs. Figure 1 shows two examples of narratives for a proverb from our dataset, along with corresponding alignment annotations. We diverge from related extant resources (Wang et al., 2020; Tan et al., 2015, 2016) on using proverbs in terms of quality of narratives, direct supervision, and having fine-grained alignment annotations. 1 We explore three tasks: (1) proverb and alignment prediction ( 5.1), (2) narrative generation for a given proverb and a set of keywords specifying a topic ( 5.2), and (3) discovering narratives with similar motifs ( 5.3). By benchmarking several LLMs, we find that existing models struggle with these tasks, suggesting much scope of improvement in abstract reasoning. In particular, humans show much higher performance in many cases. In 3, we describe the crowdsourced creation of the ePiC dataset. In 4, we analyze lexical overlap, biases, and narrative quality in ePiC . 5 describes the three tasks and details of experimental evaluation of LLMs for each task. We conclude with a discussion, and a statement of ethics and broader impact relevant to our work. Our contributions are:  We introduce ePiC , a high-quality dataset for employing proverbs in context. It contains multiple narratives for English proverbs and fine-grained annotation of aligned spans between them. We design three challenging tasks that require models to go beyond surface-level reasoning and provoke research towards making more socially grounded NLP systems. We benchmark the performance of several state-of-the-art large language models in our proposed tasks using our dataset. Our dataset and code are publicly available at: https://epic-benchmark.github.io 2 Related Work Prior works in figurative language understanding have explored a diverse set of topics, such as simile detection and generation (Niculae and Danescu-Niculescu-Mizil, 2014; Mpouli, 2017; Zeng et al., 2020; Chakrabarty et al., 2020), metaphor detection 1 Existing datasets are automatically created by scraping web-text, and supervision is heuristic (based on cooccurrences of proverbs and contexts) and generation (Dagan et al., 2005; Gao et al., 2018; Stowe et al., 2019, 2021; Chakrabarty et al., 2021b), pun identification (Poliak et al., 2018; Miller and Turkovic, 2016), and quote/proverb recommendation (Tan et al., 2015, 2016; Wang et al., 2020). Recent work (Chakrabarty et al., 2021a) has also focused on interpreting idioms and similes in narratives. Liu et al. (2019b) has explored recommending Chinese idioms through context-based recommendation and Zheng et al. (2019) formulated idiom recommendation as cloze-style reading comprehension task. Learning to quote has been explored based on fiction (Tan et al., 2015, 2016) and noisy social media conversations from Twitter, Red-dit or Weibo (Lee et al., 2016; Wang et al., 2020). In the most related prior work, authors explore a quote retrieval task borrowing inspiration from context based recommendation systems (Huang et al., 2012; He et al., 2010). Wang et al. (2020) formulated learning to quote as a generation task by using topic modeling (Miao et al., 2017; Wang et al., 2019c) in a sequence-to-sequence network. While previous work has considered idioms, proverbs and common phrases as quotes, we specifically work with proverbs. Compared to earlier datasets, our dataset is manually created and labeled. Further, ePiC includes fine-grained annotations aligning parts of proverb to parts of the narrative, which has significant possibilities for model training, evaluation and interpretability. creating the dataset in detail. Proverb collection: We obtained a candidate set of English proverbs by scraping websites of The Phrase Finder' 2 and WikiQuotes 3 . Next, this set was manually pruned to remove lexical variations of the same proverb. This manual curation led to a set of 250 proverbs, which we consider in the current version of our dataset. Narrative collection: In the second step, we use Amazon Mechanical Turk to collect a diverse set of narratives corresponding to each proverb. We collect 10 narratives contributed by distinct turkers for each proverb, leading to a total of 2500 proverb-narrative pairs. We also ensure that no turker contributes a large number of narratives to alleviate 2 https://www.phrases.org.uk/ 3 https://en.wikiquotes.org/wiki/ English_proverbs 3990 annotator bias (Geva et al., 2019) (where models can overfit to annotator characteristics) while encouraging diversity in writing style and content. The turkers were asked to write short realistic stories, preferably within 100 words. Additionally, to avoid surface-form biases, turkers were encouraged to minimize lexical overlap and to not mention the proverb or parts of it in the narrative. This was done so that doing well on the tasks requires a detailed understanding of the narratives rather than relying on surface-level cues. Turkers were paid 50 cents for each narrative for this task. Span alignment annotation: Next, we solicit fine-grained annotations between the narratives and the proverb in form of aligned spans. For this, we present proverb-narrative pairs to turkers asking them to find contiguous spans in the narrative which align well with contiguous spans in the proverb. Turkers could submit up to 5 pairs of aligned spans per proverb-narrative pair. These aligned spans highlight the grounding of a proverb in the narrative (see Figure 1). These annotations can help to verify the reasoning capabilities of various neural models by checking if these models are able to identify these correspondences, and add interpretability to our tasks. Turkers were paid 25 cents for each proverb-narrative pair annotation for this task. Statistics: Table 1 shows the statistics of narrative collection for the proverbs. The narrative writing task was perceived as challenging yet interesting by most turkers due to (a) not having outlines about topics for the narrative beforehand (b) requirement of low lexical overlap with the proverb. Thus, the narrative writing task had a learning curve and some of the narratives submitted initially were not included in the dataset. Table 2 shows some statistics of the dataset collected through the process described in 3. In this section, we analyze the characteristics and biases of the ePiC dataset in detail. Using n-grams : We evaluate the extent of lexical overlap between proverbs and narratives by computing common n-grams between them. Table 3 reports the average Jaccard similarity score between n-gram sets of proverbs and narratives, and the average number of common n-grams. On average, there are 1.27 unigrams common between narratives and proverbs (including stopwords). In comparison, randomly permuting assignments of proverbs for narratives yields an average unigram Jaccard similarity of 0.0211 and 1.06 common unigrams. Thus, the overlap metrics in the dataset are comparable to those between unrelated texts. To evaluate diversity among narratives corresponding to a proverb, we compute average Jaccard similarity between sets of unigrams for the narratives. This score is 0.107, which is comparable to a value of 0.098 for unigram overlap between pairs of narratives from different proverbs. This suggests a high lexical diversity between narratives. Using distributional embeddings : We explore if we can retrieve the correct proverb corresponding to a narrative only by using similarity in their distributional representations. The similarity between a proverb and a narrative is defined as the cosine similarity between the representation of the proverb and the narrative obtained using word2vec embeddings (Mikolov et al., 2013) or contextual embeddings from LLMs. Details of implementation are provided in Appendix F.1. For this retrieval task, we report the accuracy and Mean Reciprocal Rank of the correct proverb in 3991 LLM ACC . Table 4. We note that while all models perform better than random (with Sentence-BERT performing the best), the performance is very low when using out-of-the-box representations. In 5, we explore learning-based methods for the same setup. Diversity of narrative events: Fig 2 shows distribution of events in our dataset. Following Mostafazadeh et al. (2016) we find events as the hyponyms of the word event' or process' using WordNet (Fellbaum, 2010). We see that the top events comprise less than 3% of all events in our dataset, and the long tail of less frequent events shows the diversity of the dataset. Sentiment analysis: To evaluate the presence of sentiment association bias between proverbs and corresponding narratives (e.g., if negative sentiment proverbs only correspond to negative sentiments in narratives), we perform sentiment analysis of the narratives using VADER (Hutto and Gilbert, 2014). Figure 3 shows the average sentiment scores of the narratives corresponding to a proverb plotted against the sentiment score of the proverb. We find that the narratives are diverse in terms of their senti-C RITERION ePiC [1] [2] Relatedness 3.91 3.15 3.92 Interesting/Creative 3.57 3.34 3.63 Fluency 3.98 3.23 3.80 Overall 3.68 3.15 3.66 Table 5: Averaged Likert scale ratings for data quality. ment polarities showing a weak positive correlation (Pearson correlation score 0.35) with the sentiment score of the proverbs. Figure 4 shows the variance Sentiment of proverb A v g . in terms of the number of positive and negative sentiment narratives (out of 10) for each proverb, showing a diverse spread of narrative sentiment polarities across proverbs. For additional details, please refer to Appendix A. We perform a few additional analyses on our dataset and found that (1) around 61% of mentions in the narratives were male, (2) diverse spread of reading complexity values in narratives measured using Fleisch reading ease 4 , and (3) absence of any hate speech in the narratives of our dataset. The detailed experiments for these analyses are given in Appendix A. We perform a human evaluation of the narratives in our dataset on various criteria to judge the quality of our dataset. We perform this evaluation using the AMT platform. We randomly sample 250 proverb-narrative pairs and ask the turkers to evaluate the narratives on the following criteria:  Relatedness: how closely the narrative reflects 4 https://en.wikipedia.org/wiki/Flesch_ Kincaid_readability_tests 3992 Proverbs C oun t -10 -5 0 5 10 50 100 150 200 250 # negative # positive Figure 4: Count of narratives with positive or negative VADER sentiment for each proverb. the meaning of the proverb (1: totally unrelated, 5: perfectly related)  Interesting/Creative: how much is the narrative like a short creative or interesting story (1: very uninteresting/boring, 5: very creative/story-like)  Fluency: grammatical correctness of the narrative (1: poor English with grammatical mistakes, 5: perfect English with no errors in writing)  Overall rating All the ratings are done on Likert scales from 1 to 5, where 1 is the lowest value for each criterion and 5 is the highest. Also, the rating value 3' was calibrated to be slightly leaning to the higher end of the scale (instead of neutral) so that the turkers take a clear stand on the polarity of each criterion. Table 5 shows the qualitative evaluation of our dataset. The average overall rating was 3.67 and the average pair-wise inter-annotator agreement for labeling a narrative as overall good vs overall poor (overall score >= 3 vs < 3) is 0.84 5 . We also rate the quality of the aligned spans in our dataset similarly on a scale of 1 to 5. The average rating of the alignment between spans was 3.91 and the average pair-wise inter-annotator agreement for alignment as good vs poor (rating >= 3 vs < 3) is 0.86 5 . Table 6 highlights the key differences between ePiC and prior work that dealt with related figurative language tasks involving quotes. Notably, ePiC exclusively deals with proverbs unlike prior work (which includes common phrases and idioms such as trust your gut") and also provides granular annotations in form of annotated spans. Also, 5 Due to label imbalance kappa statistics for inter-annotator agreement are not reliable (Feinstein and Cicchetti, 1990). Thus, we report average pairwise agreement score, i.e. how often two judges agree on a label for a sample. ePiC contains narratives crowdsourced by specifically keeping proverbs in focus, rather than obtaining them using heuristic supervision. To quantify dataset quality, we ran human evaluation similar to ePiC over (1) 200 randomly drawn samples from the Reddit" dataset of quotations in context from the Wang et al. (2020), and (2) 200 randomly drawn samples from the corpus of Tan et al. (2015). Based on average Likert scores in Table 5 we find that ePiC is (1) significantly superior (using t-test; p < 0 . 05 ) on all criteria than Wang et al. (2020), and (2) better in overall ratings than Tan et al. (2015). In this section, we introduce three tasks associated with ePiC and describe their experimental setup and benchmark results: (1) Proverb and Alignment Prediction, (2) Narrative Generation, and (3) Identifying narratives with similar motifs. In this task, the objective is to predict the correct proverb for a given narrative from the set of 250 proverbs in the dataset. The motivation of this task is to test whether language models can abstract the underlying meaning of the narratives and make an analogy with the correct proverb from a large set of proverbs. In terms of applications, this task is related to proverb recommendation, which can be useful in creative writing assistants. The task is challenging as there might be multiple proverbs loosely related to the narrative context, but not be completely consonant with subliminal themes in the narrative. An underlying assumption here is that a narrative would match well with exactly one proverb. We found this reasonable for most examples in the dataset. (2) Unseen proverbs. Seen proverbs: The set of proverbs in the train and test set are the same. We divide narratives corresponding to each proverb into train and test in 6:4 ratio. So, the train and test sets have 1500 and 1000 proverb-narrative pairs respectively. Unseen proverbs: Here, we consider 150 proverbs in the train set and the remaining 100 proverbs in the test set (6:4 split on the set of proverbs). The sets of proverbs in the train and 3993 CHARACTERISTICS Tan et al. (2015) Lee et al. (2016) Wang et al. (2020) ePiC Domain Fiction Social Media Social Media Fiction Manual curation of narratives (cid:55) (cid:55) (cid:55) (cid:51) Alignment annotation (cid:55) (cid:55) (cid:55) (cid:51) Focus on proverbs (cid:55) 6 (cid:55) (cid:51) Table 6: Comparing ePiC with prior works on learning to quote based on different characteristics of the data and the collection process. respectively (since each proverb has 10 narratives). Proverb prediction: Here we focus on only predicting the corresponding proverb for a narrative, without employing the span alignments in training or evaluation. For this, we fine-tune the retrieval models based on different LLMs previously described in 4 (details of models in Appendix F.2). To evaluate performance we consider accuracy and Mean Reciprocal Rank as metrics. Table 7 shows best proverb prediction performance on test split for seen' and unseen' proverbs 7 . RoBERTa performs the best for both the seen' and unseen' settings, and the performance for all models is consistently lower for unseen proverbs (as would be expected, since this task involves much greater gen-eralization). Further, while the performance of all models is much better than chance, even the highest performance is only 28.2%. Alignment prediction: Here we focus only on predicting an aligned span from the narrative given the narrative, proverb, and a span from the proverb as inputs. We fine-tune two large language models (BERT and RoBERTa) for this by adopting a learning framework similar to answer span prediction for SQUAD (Rajpurkar et al., 2016). The language model outputs two probability distributions corresponding to the start and end positions of a span, over the narrative tokens. We iterate over all the combinations of the start and end tokens and choose the span with maximum likelihood. For span prediction, we report token-level precision, recall, and F1. Table 8 shows the results of alignment MODELSPANP SPANR SPAN F1 BERT 0.070 0.123 0.089 RoBERTa 0.068 0.143 0.092 Table 8: Alignment prediction performance for seen proverbs using LLMs (base' versions). prediction on the seen' proverbs using BERT and RoBERTa models. We find that the performance is low for both models indicating major scope for improvements. Predicting proverbs and alignment jointly: We formulate this as multi-task learning. We extend the models from the proverb prediction task by adding a component to predict span from narrative given a span from the proverb and the narrative. The language model is thus shared across the proverb prediction and span prediction tasks. The span prediction branch predicts the start and end position of the corresponding narrative span. We jointly train the model with multi-task learning of the two tasks, i.e., proverb and alignment prediction, on the seen' proverbs data split. We report the accuracy for proverb prediction and precision, recall, and F1 for span prediction. Apart from this joint model, we also consider a pipelined baseline model which first does proverb prediction, followed by span pre-6 We did not have access to the dataset to verify this. diction if the correct proverb was predicted. Table 9 shows results for the joint model and the pipelined-baseline model. The low performance of the models indicates major scope for improvements in the individual tasks. While in principle the two tasks should benefit from joint training, we find that joint training performs worse than pipelined-baseline for both proverb and alignment prediction. Future work can explore designing better models for joint training to leverage the interdependence between proverb prediction and alignment prediction. Figure 5 shows a heatmap to study the differences in prediction accuracies of BERT and RoBERTa models. We see that RoBERTa generally outperforms BERT for many cases (in Figure 5, values in the bottom-right triangle are typically greater than the top-left). Looking into the narratives for proverbs in the test set with high accuracy (>=0.75), we think a reason for the high performance could be the presence of certain words/phrases which are synonymous to some words/phrases in the proverb (for example, presence of word group' for the proverb  birds of a feather flock together '). On the other hand, there are cases when the model is confused because of multiple topics being discussed in the narrative resulting in an incorrect prediction. For example, some narratives in the test set for the proverb  life's not all beer and skittles ' describe earning money the hard way, which confused the RoBERTa model into predicting  time is money ' for such narratives. To formulate a feasible task for humans, we frame proverb prediction as a multiple choice question (MCQ) task where for each narrative, 5 proverbs are provided as choices. The set of choices includes the correct proverb and 4 other distractor 0 25 50 75 100 RoBERTa Accuracy (%) 0 25 50 75 100 BERTA cc u r a c y ( % ) 21.6 % 14.4 % 2.4 % 0.8 % 0.0 % 8.4 % 17.2 % 10.4 % 2.8 % 0.0 % 1.6 % 4.4 % 4.8 % 3.6 % 0.4 % 0.4 % 1.2 % 0.8 % 2.4 % 0.8 % 0.0 % 0.0 % 0.4 % 0.4 % 0.8 % 0% (0%, 1%] (1%, 5%] (5%, 10%] (10%, 15%] (15%, 100%] Figure 5: Heatmap showing the percentage of proverbs with various fine-tuned BERT and RoBERTa proverb prediction accuracies (for example, more than 15% of the proverbs have RoBERTA prediction accuracy as 25% and BERT prediction accuracy as 25%). proverbs, chosen by using the fine-tuned RoBERTa model. Examples of the MCQ task and details of choosing distractors are provided in Appendix B. Table 10 shows the accuracy of the human evaluation for this MCQ task on a random sample of 100 narratives from the test split of "seen" proverbs conducted using AMT. Compared to RoBERTa, we find humans are much better at this adversarially created MCQ task. Note that the performance for RoBERTa in Table 10 and Table 7 is different, as Table 10 reports accuracy only on the random sample of narratives chosen for human evaluation. The estimate for human performance is likely an under-estimate since in many cases human subjects were unfamiliar with the meanings of some of the proverbs provided in the options and as a result, focused more on surface-level cues (details of this analysis are provided in Appendix B). The average pair-wise inter-annotator agreement between human subjects for this task was 0.73 5 . This evaluation does not take into account semantic similarity between proverbs (two proverbs might be equally suitable for the same context). To explore this, we analyze the human errors on the MCQ task and find that in only around 11% of the errors, the proverb chosen by humans is semantically similar to the annotated proverb and can also be a suitable answer to the MCQ task. Details about this analysis are given in Appendix C. Future work can consider handling semantic similarity between proverbs explicitly and devise suitable evaluation metrics. One of the important use-cases for NLP models in the creative writing domain is to use these models to generate content. We explore the task of generating narratives corresponding to a proverb and a given topic (specified as a set of keywords). We benchmark the performance of two recently proposed state-of-the-art models in text generation, T5 (Raffel et al., 2020) and BART (Lewis et al., 2020), by fine-tuning them on ePiC . We divide our dataset into train and test split under seen' and unseen' proverbs settings similar to the proverb prediction task. We consider the set of verbs and named-entities as the keywords for a narrative. We train our narrative generation model conditioned on the proverb and the keywords. Table 11 shows results for automatic evaluation of the generated narratives using BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), and recall of the keywords mentioned in the generated narrative as metrics. Examples of generated narratives are given in Appendix D. We find that BART performs better than T5 on the automatic evaluation metrics. Further, we perform human evaluation to evaluate the quality of the generated narratives in AMT by considering the same criteria (and rating semantics) employed in Section 4.3. Table 12 shows the human evaluation of generated narratives using BART and T5 when tested over seen' proverbs. Low scores for BLEU and ROUGE-L in automatic metrics and low Likert ratings of the generated narratives indicate much scope for future improvement on this task. An important aspect of language understanding is the ability to make linguistic (and narrative) analogies, i.e., identifying similarity' between narratives (e.g., identifying two narratives that are variations on the Cinderella story' theme). Here, we explore the task of identifying narrative analogy by modeling similarity' between narratives based MODELBLEU  ROUGE-L  RECALL  Seen proverbs BART 4.21 30.80 0.90 T5 2.25 27.83 0.77 Unseen proverbs BART 4.39 31.36 0.93 T5 2.34 26.61 0.75 Table 11: Automatic evaluation for narrative generation on seen' and unseen' proverbs using base' versions of LLMs. on proverbs illustrated by them. For this task, two narratives are taken to be similar if they are related to the same proverb. For this task, we use the train and test split of seen' proverbs setup in the proverb prediction task. The aim is to find similar narratives for each narrative in the test split amongst all narratives in the test split. So for each narrative, there are 3 other similar narratives (corresponding to the same proverb) in the test split (containing 1000 narratives). Modeling similarity between narratives We use the learned models in the proverb prediction task to obtain a probability distribution over the proverbs for each narrative. To model similarity, we compute the distance between the (vectors representing) two probability distributions using one of the following: (1) cosine distance; (2) Jenson-Shannon divergence; (3) L2 (Euclidean) distance; and (4) L1 (Manhattan) distance. We predict the narrative closest (in terms of distance metrics) to the input narrative as the most similar. Table 13 shows the accuracy of getting a similar narrative using different distance metrics and different fine-tuned LLMs. Using cosine or Jenson-Shannon divergence as the distance metric on the probability distribution over proverbs predicted by the RoBERTa model performs best on this task. However, the overall performance of models are still low and can be benefited by devising suitable training methods for this task. We perform an additional experiment on finding similar narratives without performing proverb prediction as an intermediate step. We use a pre-trained Sentence-BERT model to obtain representations of each narrative. For a given input narrative, we calculate the cosine distance between the Sentence-BERT representation of the input narrative and all other narratives in the test set. We predict the narrative having minimum cosine distance to the input narrative as the most similar. Using this approach we find the accuracy of identifying similar narratives as 6.6%, which is lower than most values reported in Table 13. This low value highlights the diversity between narratives and the challenge in finding analogies between narratives. We introduce ePiC , a high-quality crowdsourced dataset of narratives paired with proverbs, and a suite of challenging tasks associated with this dataset. We show that these provide a challenging testbed for evaluating abstract reasoning and analogical abilities of LLMs. Future work can explore more sophisticated mechanisms to use alignment annotations in improving the performance for proverb prediction and model interpretability. Additionally, researchers can explore conditional narrative generation through more informative prompts than using keywords. ePiC can also be extended in the future by incorporating more proverbs and adding more layers of complexity like sarcasm or adversarially creating harder narratives. Most of all, the development of similarly challenging resources and tasks can enable the possibility of socially grounded NLP systems. In 4, we note that our dataset shows considerable differences in the distribution of gender of entities (61% male vs 39% female), whereas in the real world we expect the ratios to be about equally balanced. Systems that don't account for this bias might end up performing better for narratives with male entities than with females. However, we note that narratives with male and female entities show no differences in overall length or the average number of mentions to those entities. The proverbs used in our dataset were collected from free public resources without violating intellectual property rights. We do not collect any personal information from the turkers who participated in our crowdsourced tasks. We release our dataset publicly without mentioning any personal details of turkers available automatically in AMT (such as turker IDs). The turkers were compensated fairly and the payment per task is equivalent to an hourly compensation that is greater than minimum wage (based on the median time taken by turkers). For all the crowdsourcing tasks in this work, we limited the locale of eligible turkers to the USA, Canada, and the UK. Further, to ensure good-faith turkers, we required that the approval rate of the turkers be above 97%. Our screening process has selection biases that likely over-samples narrative-writers from demographics that are over-represented on AMT (eth-nically white, college-educated, lower-to-medium income, and young) (Hitlin, 2016), and this is likely to have affected the topics and type of language usage in the collected narratives. Finally, our investigation here has focused on traditional English proverbs, even while proverbs are universal in human languages and cultures (Pen-field and Duru, 1988). This poses a real risk of the development of AI models that better understand and employ specific types of figurative language than others. Such systems are likely to be less user-friendly to users that don't belong to specific social-cultural backgrounds. To mitigate these risks, but also since proverbs are universal repositories of culture-specific knowledge, future work should extend our effort to more equitably represent the variety and diversity of human thought and cultural experiences. Our investigation here, unfortunately, does not adequately do this. As the proverb goes, the road to hell is paved with good intentions.
abstain method abstain objective result other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain result method objective result abstain abstain method abstain abstain objective objective method objective other abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method other other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method other abstain method method method abstain abstain abstain method method abstain method method abstain abstain method method method abstain abstain method method method abstain abstain abstain method abstain abstain method method abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain method method abstain abstain abstain abstain abstain