Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, humans have the ability to learn new concepts from language. Here, we explore learning zero-shot classifiers for structured data 1 purely from language from natural language explanations as supervision. For this, we introduce CLUES , a benchmark for C lassifier L earning U sing natural language E xplanation S , consisting of a range of classification tasks over structured data along with natural language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. We also introduce ExEnt , an entailment-based method for training classifiers from language explanations, which explicitly models the influence of individual explanations in making a prediction. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations. We identify key challenges in learning from explanations, addressing which can lead to progress on CLUES in the future. Our code and datasets are available at: https: //clues-benchmark.github.io . Humans have a remarkable ability to learn concepts through language (Chopra et al., 2019; Tomasello, 1999). For example, we can learn about poisonous mushrooms through an explanation like a mushroom is poisonous if it has pungent odor' . Such  Equal contribution 1 By structured data, we refer to data that can be reasonably represented using tables. This is a highly flexible format for representing a lot of real-world data (e.g., spreadsheets, traditional classification datasets in CSV format, single-table databases, as well as structured text-rich data such as emails), with a large variety in possible table schemas. an approach profoundly contrasts with the predominant paradigm of machine learning, where algorithms extract patterns by looking at scores of labeled examples of poisonous and edible mushrooms. However, it is unnatural to presume the availability of labeled examples for the heavy tail of naturally occurring concepts in the world. This work studies how models trained to learn from natural language explanations can generalize to novel tasks without access to labeled examples. While prior works in this area (Srivastava et al., 2017, 2018; Hancock et al., 2018; Murty et al., 2020; Andreas et al., 2018; Wang* et al., 2020; Ye et al., 2020; Zhou et al., 2020) have explored explanations as a source of supervision, they evaluate models on a small number of tasks (2-3 relation extraction tasks in (Hancock et al., 2018; Wang* et al., 2020; Murty et al., 2020; Zhou et al., 2020), 7 email categorization tasks (Srivastava et al., 2017)). Owing to the paucity of large-scale benchmarks for 6523 learning from explanations over diverse tasks, we develop CLUES , a benchmark of classification tasks paired with natural language explanations. Over the last few decades, researchers and engineers alike have put immense effort into constructing structured and semi-structured knowledge bases (e.g., structured tables on Wikipedia, e-commerce sites, etc.). Developing models that can reason over structured data is imperative to improve the accessibility of machine learning models, enabling even non-experts to interact with such data. Hence, in this work, we specifically formulate our classification tasks over structured data. Our benchmark is divided into CLUES-Real and CLUES-Synthetic consisting of tasks from real-world (UCI, Kaggle, and Wikipedia) and synthetic domains respectively. Explanations for CLUES-Real are crowdsourced to mimic the diversity and difficulty of human learning and pedagogy. For CLUES-Synthetic , we generate the explanations programmatically to explicitly test models' reasoning ability under a range of structural and linguistic modifications of explanations. We train models with a mix of explanations and labeled examples, in a multi-task setup, over a set of seen classification tasks to induce generalization to novel tasks, where we do not have any labeled examples. Ye et al. (2021) refer to this problem setup as cross-task generalization". Some recent methods on cross-task generalization from language use instructions/prompts (Mishra et al., 2022; Sanh et al., 2022; Wei et al., 2021) describing information about what is the task?' to query large language models. In contrast, language explanations in CLUES provide the logic for performing the classification task, or intuitively how to solve the task?' . For the running example of mushroom classification, an instruction/prompt might be can you classify a mushroom with pungent odor as poisonous or edible?' . On the other hand, an example of an explanation in CLUES is a mushroom is poisonous if it has pungent odor' . We find that simply concatenating explanations to the input does not help pre-trained models, like RoBERTa (Liu et al., 2019), generalize to new tasks. Thus, we develop ExEnt , an entailment-based model for learning classifiers guided by explanations, which explicitly models the influence of individual explanations in deciding the label of an example. ExEnt shows a relative improvement of up to 18% over other baselines on unseen tasks. To identify the challenges of learning from explanations, we perform extensive analysis over synthetic tasks. Our analysis explores how the structure of an explanation (simple clauses vs. nested clauses) and the presence of different linguistic components in explanation (conjunctions, disjunctions, and quantifiers) affect the generalization ability of models. The rest of the paper is structured as follows: we describe our crowdsourced-benchmark creation pipeline in 3. In 4, we analyze our collected data. In 5, we describe our models, experiments, and results. We conclude with a brief discussion on the contributions and our findings, followed by a statement of ethics and broader impact. Our contributions are:  We introduce CLUES , a benchmark for learning classifiers over structured data from language. We develop ExEnt , an entailment-based model for learning classifiers guided by explanations. ExEnt shows a relative improvement of up to 18% over other baselines on generalization to novel tasks. We explore the effect on the generalization ability of models learning from language by ablating the linguistic components and structure of explanations over our benchmark's synthetic tasks. Learning concepts from auxiliary information: Prior work has explored techniques to incorporate side-information' to guide models during training (Mann and McCallum, 2010; Ganchev et al., 2010). More recently, researchers have explored using language in limited data settings for learning tasks such as text classification (Srivastava et al., 2017, 2018; Hancock et al., 2018) and question answering (Wang* et al., 2020; Ye et al., 2020). However, we diverge from these works by exploring the generalization ability of classifiers learned by using language over novel tasks as opposed to gauging performance only on seen tasks. Explanation-based Datasets: The role of explanations and how they can influence model behavior is a widely studied topic in machine learning (Wiegreffe and Marasovic, 2021). Among language-based explanation studies, past work has primarily developed datasets that justify individual predictions made by a model (also called, local explanations) (Rajani et al., 2019; Camburu et al., 2018), inter alia . In contrast, our work focuses 6524 on explanations that define concepts and capture a broad range of examples rather than individual examples. Our notion of explanations is shared with Andreas et al. (2018); Srivastava et al. (2017, 2018). We differ from these works as (1) our benchmark comprises a large set of classification tasks spanning diverse concepts for learning from explanations as opposed to working on a limited set of tasks in prior work and (2) our benchmark is domain agnostic in the source of classification tasks considered as long as we can represent the inputs of the task in a tabular (structured) format. Few-shot & Zero-shot learning: Large pretrained language models (LMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) have been shown to perform impressively well in few-shot settings (Brown et al., 2020; Lester et al., 2021). Reformulating natural language tasks with patterns has been shown to boost few-shot learning ability for small language models as well (Schick and Schtze, 2021; Tam et al., 2021). More recently, a few works have focused on evaluating the generalization of models to unseen tasks by using prompts and performing multi-task training (Mishra et al., 2022; Ye et al., 2021; Sanh et al., 2022; Min et al., 2021; Chen et al., 2022; Aghajanyan et al., 2021) While the training and evaluation setup is similar, our work is significantly different from these works as (1) the explanations in our work provide rationales for making a classification decision as opposed to explaining a task using prompts, (2) we explore classification over structured data as opposed to free-form text by designing a model that can leverage explanations. In this section, we describe our benchmark creation process in detail. In CLUES , we frame classification tasks over structured data represented in tabular format. Based on the source of tables used to construct the classification tasks, we consider two splits of our benchmark, CLUES-Real (real-world datasets) and CLUES-Synthetic (synthetic datasets). We first gather/create classification tasks from UCI, Kaggle, and Wikipedia tables, then collect explanations for each classification task. Classification tasks from UCI and Kaggle. UCI ML repository 2 and Kaggle 3 host numerous datasets for machine learning tasks. For our benchmark, we pick out the tabular classification datasets. Then, we manually filter the available datasets to avoid ones with (a) many missing attributes and (b) complex attribute names that require extensive domain knowledge making them unsuitable for learning purely from language. CLUES-Real contains 18 classification tasks from UCI and 7 from Kaggle (the details of tasks are in Appendix B). Mining tables from Wikipedia. Wikipedia is a rich, free source of information readily accessible on the web. Further, a lot of this information is stored in a structured format as tables. We explore creating additional classification tasks based on tables from Wikipedia, where each row in a table is assigned a category label. However, only a small fraction of the tables might be suitable to frame a classification task for our benchmark. Thus, we need to identify suitable tables by mining a large collection of tables from Wikipedia (we use Wikipedia dump available on April 2021). We formalize this mining-and-pruning process as a crowdsourcing task (on Amazon Mechanical Turk), where we present each turker with a batch of 200 tables and ask them to pick out suitable tables from that batch. For a table considered suitable by a turker, we further ask the turker to mention which column of the table should be considered as providing the classification labels. We identified 11 classification tasks corresponding to 9 Wikipedia tables after mining around 10K Wikipedia tables (the details of tasks are provided in Appendix B). Our explanation collection process consists of two stages  (1) teachers providing explanations after reviewing multiple labeled examples of the task, and (2) students verifying explanations and classifying new examples based on explanations for the tasks. Collecting explanations : We use the Amazon Mechanical Turk (AMT) platform to collect explanations for CLUES-Real . In each HIT, we provide turkers with a few labeled examples of a dummy task (each corresponding to a row in a table) and a set of good and bad explanations for the task to 2 https://archive.ics.uci.edu/ml/ 3 https://www.kaggle.com/datasets 6525 teach them about the expected nature of explanations. Next, we test them on a qualification quiz' to gauge their understanding of good explanations. Upon qualification, the turker advances to the explanation collection phase of the HIT. At this stage, the turker is provided with 15-16 labeled examples of a task in CLUES-Real and we ask them to write explanations describing the logic behind the classification for each class. Turkers are required to submit a minimum of two explanations (  5 tokens each) for each task. Further, teachers can test their understanding by taking a validation quiz, where they make predictions over new unlabeled examples from the task. Based on their informed classification accuracy, teachers can optionally refine their explanations. Finally, when turkers are content with their performance, they freeze' the explanations and advance to the test-quiz where they are evaluated on a new set of unlabeled examples from the task (dif-ferent from validation quiz). 4 We will refer to turkers who have provided responses at this stage as teachers' since they provide explanations to teach' models about different classification tasks. Verification of explanations : After the explanation collection, we validate the utility of the sets of explanations for a task from each teacher by evaluating if they are useful they are for other humans in learning the task. For this, a second set of turkers 5 is provided access to the collected explanations from a teacher for a task, but no labeled examples. These turkers are then asked to predict the labels of test examples from the held-out test set, solely based on the provided explanations. Additionally, we ask turkers in the verification stage to give a Likert rating (1-4 scale) on the usefulness of each explanation. Since the turkers in the verification stage perform the classification task using language explanations from a teacher, we refer to them as students' for our setup. Thus, the tasks in CLUES-Real contain explanations from multiple teachers and multiple students corresponding to a teacher. This provides rich information about variance in teacher and student performance indicating how amenable different tasks are for learning via language. We provide insights into the performance of teachers and students of our setup in 4. The complexity and fuzziness of real-world concepts and the inherent linguistic complexity of crowdsourced explanations can often shroud the aspects of the task that make it challenging for models to learn from explanations. To evaluate models in controlled settings where such aspects are not conflated, we create CLUES-Synthetic , a set of programmatically created classification tasks with varying complexity of explanations (in terms of structure and presence of quantifiers, conjunctions, etc.) and concept definitions. We create tasks in CLUES-Synthetic by first selecting a table schema from a pre-defined set of schemas, then generating individual examples of the task by randomly choosing values (within a pre-defined range, obtained from schema) for each column of the table. Next, we assign labels to each example by using a set of rules' for each task. In this context, a rule' is a conditional statement (analogous to conditional explanations that we see for real-world tasks) used for labeling the examples. We use the following types of rules that differ in structure and complexity ( c i denotes i th clause and l denotes a label):  Simple: IF c 1 THEN l  Conjunctive: IF c 1 AND c 2 THEN l  Disjunctive: IF c 1 OR c 2 THEN l  Nested disjunction over conjunction: IF c 1 OR ( c 2 AND c 3 ) THEN l  Nested conjunction over disjunction: IF c 1 AND ( c 2 OR c 3 ) THEN l  For each of the above, we include variants with negations (in clauses and/or labels): Some examples IF c 1 THEN NOT l , IF c 1 OR NOT c 2 THEN l We also consider other linguistic variations of rules by inserting quantifiers (such as always', likely'). The synthetic explanations are template-generated based on the structure of the rules used in creating 6526 Vocabulary 1026 Avg. the task. For brevity, we defer additional details on the use of quantifiers, label assignment using rules, and creation of synthetic explanations to Appendix A. Overall we have 48 different task types (based on the number of classes and rule variants) using which we synthetically create 144 classification tasks (each containing 1000 labeled examples). Task Statistics : Table 1 shows the statistics of tasks in CLUES . The real-world tasks in our benchmark are from a wide range of domains, such as data corresponding to a simple game (e.g. tic-tac-toe), medical datasets (e.g. identifying liver pa-tients), merit-classification of teachers and students, network-related datasets (eg. internet-firewall), among others. The synthetic tasks are created using table schemas denoting different domains, such as species of animals, species of birds, etc. (details in Appendix A). As seen in Table 1, 5.4 explanation sets were collected for each classification task from human teachers on average. Further, each explanation set was verified by 3 students during the verification task. An aggregate of 133 teachers provide 318 explanations for tasks in CLUES-Real . All collected explanations were manually filtered and irrelevant explanations were removed. Lexical analysis of explanations : Table 2a shows the statistics for explanation texts in our dataset. 6 We evaluate the average length of the explanation texts, vocabulary size and number of unique bigrams present in the explanations. Explanation characteristics : Following Chopra et al. (2019), we categorize the explanations based on the different aspects of language (generics, quantifiers, conditional, and negation) present in these explanations. Table 3 shows the statistics of various categories in our dataset. Note that an explanation might belong to more than one category (for example, an example like if the number of hands equal 6 Statistics in Table 2a was obtained using the spacy tok-enizer. CATEGORYEXAMPLEREALSYN Generic Being over 50 increases the risk of a stroke. 48 % 50 % Quantifier ... usually means you won't have heart disease. 52 % 50 % Conditional If color code ... , then ... 15 % 100 % Negations ... is not low. 16 % 50% Table 3: Count of explanations in our dataset based on various aspects of language present in them to 2, then it is usually foo" , will be categorized both as having both conditional and quantifiers.) We found that around 52% of the explanations for the real-world tasks had quantifiers (such as some', majority', most', etc.) in them. A full list of quantifiers present in the data is given in Appendix A. Reading complexity : We analyze the reading complexity of crowdsourced explanations by using Flesch reading ease 7 . Reading complexity values for our crowdsourced explanations vary from 3.12 (professional grade reading level) to 106.67 (easier than 3rd-grade reading level), with a median value of 65.73 (8th/9th-grade reading level). Usefulness of the explanations : During the validation stage, we ask the turkers to provide a rating (on a Likert scale from 1 to 4) on the utility of the explanations for classification. The semantics of ratings are, 1  not helpful', 2  seems useful', 3  helped in predicting for 1 sample', and 4  mostly helpful in prediction'. The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful, even if they did not directly help predict labels in some cases. In Figure 2(a), we also provide a histogram of the Likert ratings provided by the students. Characteristics of teachers and students : Figure 2(b) shows the normalized teacher performance vs normalized student performance for teacher-student pairs in CLUES-Real . Normalized performance of an individual teacher (or, student) on a task is defined as the difference between the performances of the teacher (or, student) and an average teacher (or, student) for the same task. The positive correlation (  = 0.17) suggests that students tend 7 https://en.wikipedia.org/wiki/Flesch_ Kincaid_readability_tests 6527 1 2 3 4 Rating 0 100 200 300 F r e q u e n c y Likert Ratings of Explanations to perform well if taught by well-performing teachers. Positive correlation (  = 0.48) in Figure 2(c), indicates that task difficulty (captured by classification accuracy) is well-correlated for a teacher and student on average. On visualizing the difference between an average student and an average teacher performance for each task in CLUES-Real , we find that an average teacher performs better than the average student on most tasks. However, for the tic-tac-toe' task in CLUES-Real , we find that the student accuracy was around 13% higher than average teacher performance. We hypothesize that this task can be solved by commonsense reasoning without relying on the provided explanations, resulting in students performing better than teachers. We quantify the average performance of teachers and students on CLUES-Real in Table 4. 8 We find that students per-8 Note that teacher scores in the tables and figures do not include 9 Wikipedia Tasks for which the authors formed the form lower than teachers on average as expected since a teacher has more expertise in the task. Moreover, it is challenging to teach a task perfectly using explanations in a non-interactive setting where a student cannot seek clarifications. Additional data analysis and details of HIT compensation can be found in Appendix C and D. 5 Experiment Setup and Models In this section, we describe our training and evaluation setup, our models, and experimental findings. Our goal is to learn a model that, at inference, can perform classification over an input x to obtain the class label y , given the set of explanations E for the classification task. Figure 4 shows our setup, where we train our model using multi-task training over a set of tasks T seen and evaluate generalization to a new task, t  T novel . The task split we use for our experiments can be found in Appendix E.1. We select our best model for zero-shot evaluation based on the validation scores on the seen tasks. Since we do not make use of any data from the novel tasks to select our best model, we maintain the true zero-shot setting (Perez et al., 2021). We encode each structured data example, x , as a text sequence, by linearizing it as a sequence of attribute-name and attribute-value pairs, separated by [SEP] tokens. To explain, the leftmost attribute-name and attribute-value pair of structured input example in Figure 1 is represented as odor | pungent' . The linearization allows us to make use of pre-trained language models for the classification task. Our linearization technique explanations. is similar to the one used in Yin et al. (2020) with the exception that we do not use the column type. We will refer to the linearized format of structured inputs by Features-as-Text' or FaT'. For our baselines, we make use of a pre-trained RoBERTa model (Liu et al., 2019). However, RoBERTa with the standard-fine-tuning approach cannot allow a generalization test as the number of output classes varies for each task. Furthermore, we cannot train individual class heads at inference since we test zero-shot . Hence, we make the following modifications to make RoBERTa amenable for zero-shot generalization tests: a pre-trained RoBERTa model takes the linearized structured data (FaT) as input and outputs a representation for this context (in the [CLS] token). Next, we run another forward pass using RoBERTa to obtain a representation of the labels based on their text (e.g., poisonous' or edible' for our example in Figure 1). Finally, we compute the probability distribution over labels by doing a dot-product of the representations of the input and the labels. We train this model using cross-entropy loss. In our experiments, we refer to this model as RoBERTa w/o Exp since the model does not use any explanations. We also experiment with a RoBERTa w/ Exp. model where a RoBERTa model takes as input a concatenated sequence of all the explanations for the task along with FaT. The rest of the training setup remains the same as RoBERTa w/o Exp. We find that a simple concatenation of explanations is not helpful for zero-shot generalization to novel tasks (results in Figure 6). Next, we describe ExEnt which explicitly models the role of each explanation in predicting the label for an example. To model the influence of an explanation towards deciding a class label, we draw analogies with the entailment of an explanation towards the structured input. Here, given a structured input ( premise ) and an explanation ( hypothesis ), we need to decide whether the explanation strengthens the belief about a specific label ( entailment ), weakens belief about a specific label ( contradiction ) or provides no information about a label ( neutral ). Figure 5 shows the overview of our explanation-guided classification model, ExEnt ; given a structured input and explanation of a task, let l exp denote the label mentioned in the explanation, and L denote the set of labels of the task. The entailment model assigns logits p e , p c and p n to the hypothesis being entailed, contradicted or neutral respectively w.r.t. the premise. Based on the label assignment referred to by an explanation, we assign logits to class labels as follows:  If explanation mentions to assign a label : Assign p e to l exp , p c is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L . If explanation mentions to not assign a label : This occurs if a negation is associated with l exp . Assign p c to l exp , p e is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L . We obtain logit scores over labels of the task corresponding to each explanation as described above. We compute the final label logits by aggregating (using mean) over the label logits corresponding to each explanation of the task. The final label logits are converted to a probability distribution over labels, and we train ExEnt using cross-entropy loss. In experiments, we consider a pre-trained RoBERTa model fine-tuned on MNLI (Williams et al., 2017) corpus as our base entailment model. 9 Further, in order to perform the assignment of logits using an explanation, we maintain meta-information for each explanation to (1) determine if the explanation mentions to assign' a label or not assign' a label, and (2) track l exp (label mentioned in explanation). For CLUES-Synthetic , we parse the templated explanations to obtain the 9 Weights link: https://huggingface.co/ textattack/roberta-base-MNLI 6529 MLM Exp. meta-information, while for the explanations in CLUES-Real , the authors manually annotate this meta-information. Additional training details and hyperparameters are provided in Appendix E. 5.4 Zero-Shot Generalization Performance We evaluate ExEnt and the baselines on zero-shot generalization to novel tasks in our benchmark as described in 5.1. We train separate models for CLUES-Real and CLUES-Synthetic . Figure 6 shows the generalization performance of all models. On CLUES , we find that ExEnt outperforms the baselines suggesting that performing entailment as an intermediate step helps aggregate information from multiple explanations better. On CLUES-Real , ExEnt gets an 18% relative improvement over the baselines while having an 11% relative improvement on CLUES-Synthetic To evaluate the utility of our synthetic tasks in enabling transfer learning to real-world tasks, we finetune a ExEnt model pre-trained on synthetic tasks. We experiment with three pre-training task sets -CLUES-Synthetic , CLUES-Synthetic (3x) and CLUES-Synthetic (5x) consisting of 144, 432, and 720 tasks. These larger synthetic task sets are created by sampling tasks from each of the 48 different synthetic tasks types similar to how CLUES-Synthetic was created (see 3.2 for refer-ence). We find that pre-training on synthetic tasks boosts the performance of ExEnt on the novel tasks of CLUES-Real by up to 39% (relative) over the RoBERTa w/o Exp. model. Human Performance To situate the performance of the automated models, we performed human evaluation for tasks in test split of CLUES-Real using AMT. For this, we sampled at most 50 examples 10 from the test split of tasks in CLUES-Real and each example was labeled' by 2 turkers using the explanations of the best teacher' (the teacher whose students got the best performance during explanation verification' stage; see 3.1.2 for reference). The average human accuracy for this was about 70%. However, the performance numbers of humans and models are not directly comparable as the model looks at all the explanations for the task, whereas the humans observe a small number of explanations. Humans also see multiple examples of the task during the evaluation, which they can use to fine-tune their understanding of a concept. The automated models don't have a mechanism to leverage such data. To identify key challenges in learning from explanations, we perform experiments ablating the linguistic components and structure of explanations. For a robust analysis, we generate more tasks for each task type in CLUES-Synthetic , making 100 tasks for each of the 48 different task-types in CLUES-Synthetic (axes of variation include 4 negation types, 3 conjunction/disjunction types, 2 10 Many tasks (such as tasks created from Wikipedia tables) have less than 50 examples in their test split.) 6530 noneg. Appendix A.5). We evaluate the generalization performance of ExEnt to novel tasks on each of the different types separately by training separate models for each task type. Figure 7 shows the relative gain in generalization performance of models learned using explanations compared to the performance of baseline RoBERTa w/o Exp. 11 Our results indicate that learning from explanations containing quantifiers is highly challenging. In the presence of quantifiers, models guided by explanations perform on par with the baseline RoBERTa w/o Exp model. Negations also pose a challenge, as indicated by the decline in relative gains of models guided by explanation compared to the RoBERTa w/o Exp model. Structurally complex explanations (containing conjunc-tions/disjunctions of clauses) are also hard to learn from compared to simple conditional statements. These challenges provide a fertile ground for future research and improvements. We have introduced CLUES , a benchmark with diverse classification tasks over structured data along with natural language explanations to learn them. CLUES is agnostic in the domain of tasks allowing the research community to contribute more tasks in the future. We also present ExEnt , an entailment-based model to learn classifiers guided by explanations. Our results are promising and indicate that explicitly modeling the role of each explanation through entailment can enable learning classifiers for new tasks from explanations alone. Future work can explore the open challenges in learning from explanations, such as modeling the influence of quantifiers and negations present in an explanation. Our empirical analyses here aggregates explana-11 Accuracies have been averaged over the multi-class and binary datasets since the trends remain the same across both. tions for a task from multiple teachers. Future work can explore learning from explanations from individual teachers, as well as cross-teacher variance. Alternatively, rather than treat explanations from different teachers homogeneously, future work can model trustworthiness of a crowd of teachers from their provided explanations. All tables in CLUES-Real were collected from free public resources (with required attributions) and tables in CLUES-Synthetic were created by us programmatically. We do not collect any personal information from the turkers who participated in our crowdsourced tasks. The dataset has been released without mentioning any personal details of turkers available automatically in AMT (such as turker IDs). The turkers were compensated fairly and the payment per task is equivalent to an hourly compensation that is greater than minimum wage (based on the median time taken by turkers). We provide details of the reward structure for the crowdsourcing tasks in Appendix D. For the Wikipedia mining task in this work, we limited the locale of eligible turkers to US, UK, New Zealand and Australia. For other crowdsourcing tasks, we limited the locale of eligible turkers to US. Further, to ensure good-faith turkers, we required that the approval rate of the turkers be above 98%. Our screening process has selection biases that likely over-samples turkers from demographics that are over-represented on AMT (ethnically white, college-educated, lower-to-medium income and young) (Hitlin, 2016), and this is likely to affect the type of language usage in the collected explanations. The broader impact of this research in the longer term could make developing predictive technologies more accessible to ordinary users, rather than data-scientists and experts alone.
abstain abstain objective abstain abstain abstain abstain abstain method other abstain abstain objective abstain abstain abstain objective other objective abstain abstain method method abstain method objective abstain abstain abstain abstain abstain abstain objective abstain method objective method method result objective objective objective objective objective other other objective other other objective abstain objective other other other abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method other abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain method abstain method abstain method abstain other abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain method abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method method abstain method method method method method abstain method method abstain method method method method abstain abstain method abstain method abstain abstain method abstain abstain abstain method method method method other abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain abstain method method abstain abstain method method abstain abstain abstain