For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know how best to select text sources to collect a variety of challenging examples. In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples. To our surprise, we find that passage source, length, and readability measures do not significantly affect question difficulty. Through our manual annotation of seven reasoning types, we observe several trends between passage sources and reasoning types, e.g., logical reasoning is more often required in questions written for technical passages. These results suggest that when creating a new benchmark dataset, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority. State-of-the-art systems have shown performance comparable with humans on many recent natural language understanding (NLU) datasets (Devlin et al., 2019; Sun et al., 2021), suggesting that these benchmarks will no longer be able to measure future progress. To move beyond this, we will need to find better ways of building difficult datasets, ideally without sacrificing diversity or coverage (Bowman and Dahl, 2021). To obtain such human-written examples at scale, there are active lines of crowdsourcing research on protocols of worker handling and feedback (Nangia et al., 2021) and the design of the collection task (Ning et al., 2020; Rogers et al., 2020). However, we do not have clear MCTest : Tony walked home from school on his birthday. He was surprised to see a lot of cars in front of his house. When he opened the door and entered the house, he heard a lot of people yell, Surprise! It was a surprise party for his birthday. His parents called all his friends' parents and invited them to come to a party for Tony. [...] Q: Who were invited to the party and by who? (cid:3) Tony's parents invited only his friends (cid:3) Tony invited his friends and their parents (cid:3) Tony's parents invited his friends' parents (cid:88)(cid:3) Tony's parents invited his friends and their parents ReClor : Humanitarian considerations aside, sheer economics dictates that country X should institute, as country Y has done, a nationwide system of air and ground transportation for conveying seriously injured persons to specialized trauma centers. Timely access to the kind of medical care that only specialized centers can provide could save the lives of many people. [...] Q: What is the economic argument supporting the idea of a transportation system across the nation of Country X? (cid:3) Building the transportation system creates a substantial increase of jobs for the locals (cid:88)(cid:3) Increasing access to specialized medical centers can lower the chance of the workforce population dying (cid:3) Transportation ticket prices directly contribute to the government's revenue (cid:3) Country Y was successful with their attempts to potentially save lives so Country X should try it as well Figure 1: Example questions for passages from simple narratives (MCTest) and technical arguments (ReClor). Crowdsourced datasets in reading comprehension use passages taken from a variety of sources, such as news articles, exams, and blogs, about which questions are written (Lai et al., 2017; Trischler et al., 2017; Rogers et al., 2020). The first example in Figure 1 is from MCTest (Richard-son et al., 2013), the passages of which are written in grade-school-level English. The second example is from ReClor (Yu et al., 2020), which consists of passages and questions written for graduate and law school admission examinations. We hypothesize that difficult passages, such as those in the second example, are more suitable for crowdsourcing challenging questions. Passages that are linguistically 6951 complex and have dense information could help facilitate the writing of questions that require understanding a wide range of linguistic and world knowledge, following intricate events, and comprehending logical arguments. In contrast, easy passages, as in children's stories, likely talk about common situations and simple facts, which might prevent workers from writing difficult questions. In this work, we crowdsource multiple-choice reading comprehension questions to analyze how question difficulty and type are affected by the choice of source passage. Using passages extracted from seven different sources, we ask crowdworkers to write questions about the given passages. We compute the difference between human and machine accuracy, using it as a measure of the question difficulty, to investigate whether there is a correlation between the question difficulty and linguistic aspects of the passage, such as their source, length, and readability. In addition to a standard setting where we directly accept crowdworkers' submissions, we use an adversarial setting in which they have to write questions that fool a strong reading comprehension model (Bartolo et al., 2020; Kiela et al., 2021). Previous work finds that questions that require numerical reasoning frequently appear in the adversarial data collection of the extractive QA task on Wikipedia articles (Kaushik et al., 2021), but our aim is to see whether we observe a similar trend in multiple-choice questions written for different passage sources or if the adversarial setting is useful for collecting especially diverse questions. To our surprise, we find that the difficulty of collected questions does not depend on the differences of passages in linguistic aspects such as passage source, passage length, FleschKincaid grade level (Kincaid et al., 1975), syntactic and lexical surprisal, elapsed time for answering, and the average word frequency in a passage. Our main positive finding comes through our manual annotation of the types of reasoning that each question targets, where we observe that questions that require numerical reasoning and logical reasoning are relatively difficult. In addition, we find several trends between the passage sources and reasoning types. For example, logical reasoning is more often required in questions written for technical passages, whereas understanding of a given passage's gestalt and the author's attitude toward it are more frequently required for argumentative and subjective passages than expository passages. These results suggest that when creating a new benchmark dataset or choosing one for evaluating NLU systems, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority. Our collected datasets could be useful for training reading comprehension models and for further analysis of requisite knowledge and comprehension types in answering challenging multiple-choice questions. 1 2 Related Work Crowdsourcing NLU Datasets Crowdsourcing has been widely used to collect human-written examples at scale (Rajpurkar et al., 2016; Trischler et al., 2017). Crowdworkers are usually asked to write questions about a given text, sometimes with constraints imposed to obtain questions that require specific reasoning skills such as multi-hop reasoning (Yang et al., 2018) or understanding of temporal order, coreference, or causality (Rogers et al., 2020). In this study, to analyze naturally written examples, we do not consider specific constraints on questions or answer options. Current benchmark datasets constructed by crowdsourcing may not be of sufficient quality to precisely evaluate human-level NLU. For example, Ribeiro et al. (2020) reveal that state-of-the-art models in traditional NLP benchmarks fail simple behavioral tests of linguistic capabilities ( checklists ). Chen and Durrett (2019) and Min et al. (2019) show that questions in multi-hop reasoning datasets such as HotpotQA by Yang et al. (2018) do not necessarily require multi-hop reasoning across multiple paragraphs. To investigate how to collect high-quality, challenging questions through crowdsourcing, Nangia et al. (2021) compare different sourcing protocols and find that training workers and providing feedback about their submissions improve the difficulty and quality of their reading comprehension questions. To encourage workers to write difficult examples, Bartolo et al. (2020) propose to collect questions using a model-in-the-loop setting. Although this adversarial approach enables us to collect challenging questions efficiently, Gardner et al. (2020) point out that the collected examples might be bi-1 Our datasets, annotation instructions and results, and crowdsourcing scripts are available at https://github. ased towards the quirks of the adversary models. Bowman and Dahl (2021) extend this argument, and point out that adversarial methods can systematically eliminate coverage of some phenomena. This is also supported by Kaushik et al. (2021), but their findings are limited to extractive QA for Wikipedia articles. Our motivation is to see if this argument is applicable to the multiple-choice format with a wide range of passage sources for which we expect crowdworkers to write linguistically diverse questions and answer options. Sources of NLU Datasets Reading comprehension datasets are often constructed with a limited number of passage sources. Rajpurkar et al. (2016) sample about five hundred articles from the top 10,000 articles in PageRank of Wikipedia. Similarly, Dua et al. (2019) curate passages from Wikipedia articles containing numeric values to collect questions for mathematical and symbolic reasoning. Khashabi et al. (2018) construct a dataset in which questions are written for various passage sources such as news articles, science textbooks, and narratives. However, we cannot use their questions for our analysis of the variation of naturally written questions because they are designed to require local multi-sentence reasoning (such as coref-erence resolution and paraphrasing) by filtering out questions answerable only with a single sentence. Similarly to our work, Sugawara et al. (2017) find that readability metrics and question difficulty do not correlate in reading comprehension datasets. Our study differs in the following two points, which could cause different findings: First, their observational study of existing datasets has fundamental confounding factors because the questions they examine are constructed using different sourcing methods (e.g., automatic generation, expert writing, and crowdsourcing), which could have an impact on the question difficulty. We aim to investigate uniformly crowdsourced examples across seven different sources to obtain insights for future data construction research using crowdsourcing. Second, they define question difficulty using human annotations alone, but this does not necessarily reflect the difficulty for current state-of-the-art models. In this study, we define the question difficulty as the humanmachine performance gap using eight recent strong models, which enables a more fine-grained analysis of the collected questions for a better benchmark of current models. consisting of different in-domain and out-domain datasets. However, they combine datasets in different task formats and sourcing methods, which prevents us from comparing questions across passage sources alone. In contrast, our focus is to compare questions collected by crowdsourcing for the same task format to analyze the question difficulty for current state-of-the-art models. We adopt the multiple-choice format because, as discussed by Huang et al. (2019), it allows us to evaluate both human and machine performance easily. This study aims to analyze what kinds of passages make crowdsourced reading comprehension questions difficult. We use Amazon Mechanical Turk. To collect difficult and high-quality examples, we require crowdworkers to take a qualification test before accepting our question writing and validation tasks. The qualification test has two parts, which we run in separate tasks: question answering and writing. To take the qualification test, workers have to meet the following minimum qualifications: based in the United States, Canada, or United Kingdom, have an approval rate of at least 98%, and have at least 1,000 approved tasks. The question answering task is used to identify workers who answer reading comprehension questions carefully. A single question answering task has five questions that are randomly sampled from the validation set of ReClor in which most questions are taken from actual exams. Those who correctly answer at least four out of the five questions proceed to the next qualification phase. The question writing task is used to familiarize workers with the writing of multiple-choice reading comprehension questions and select those who can carefully write examples. We ask workers to write two questions given two different passages randomly sampled from the validation set of RACE (Lai et al., 2017). This dataset consists of self-contained passages written for middleand highschool exams in various subjects, which we expect the workers to be able to write questions for easily. Following Nangia et al. (2021), we then review the workers' submissions and grade them using a rubric with four criteria: the question (1) is answerable without ambiguity ( yes or no ); (2) requires 6953 reading the whole passage (five-point scale); (3) is creative and non-obvious (five-point scale); and (4) has distractor answers that could look correct to someone who has not read the passage carefully ( more than one , one , or no ). We rank workers using this rubric and allow approximately the top 50% of workers to proceed to the main writing task. We make sure that these workers write two unambiguous and answerable questions. In the main writing task, a worker is shown a single passage and asked to write a question about it along with four answer options. We provide instructions where we describe that questions have to be challenging but still answerable and unambiguous for humans, and we include good and bad examples to illustrate what kinds of questions we aim to collect. For example, good examples require reading the whole passage and ask about characters' motivations or consequences of described events, while bad examples only ask about a simple fact or are answerable without reading the passage (Ap-pendix P). Each worker who passes the qualification round is randomly assigned to either standard or adversarial data collection. In the standard collection, we accept workers' submissions without any filtering. In the adversarial collection, a written question is sent to a reading comprehension model immediately. If the model cannot answer that question correctly, we accept it. We allow workers to submit questions (i.e., get paid) after three attempts even if they keep failing to fool the model. We use UnifiedQA 3B v2 (Khashabi et al., 2020) for the adversary model, which is trained on a wide variety of question answering datasets such as MCTest, RACE, NarrativeQA (Kocisk et al., 2018), and SQuAD. While the source of training data that we use in our models will inevitably influence our findings, focusing on a model with very diverse pretraining and fine-tuning will minimize this effect. Passage Sources We use passages from the following seven sources: (1) MCTest children's narratives, (2) Project Gutenberg narratives, (3) Slate online magazine articles from the 1990s sourced from the Open American National Corpus (Ide and Suderman, 2006), (4) middleand high-school exams from RACE, (5) graduate-level exams from ReClor, and (6) science and (7) arts articles from Wikipedia. We use the passages from the training sets of MCTest, RACE, and ReClor. For Gutenberg, Slate, and Wikipedia, we split available books and articles into passages. Details are in Appendix A. In the writing task, a passage is randomly taken from a passage pool in which there are the same number of passages extracted from each source. We collect the votes of five workers for each of the collected questions. Those workers who passed the question answering task of the qualification round can accept the validation tasks. To incen-tivize workers, we use preexisting gold-labeled examples (from Nangia et al., 2021) as catch trials, representing about 10% of the tasks, and pay a bonus of $0.50 USD if a worker can answer those questions correctly at least 80% of the time. If a worker fails to answer them at least 60% of the time, we disqualify the worker from future rounds of data collection. Worker Pay and Logistics For the writing tasks, the base pay is $2.00 per question, which we estimate to be approximately $15.00 per hour based on measurements from our pilot runs. If a worker succeeds in fooling the model in adversarial data collection, they receive an additional bonus of $1.00. For validation, a single task consisting of five questions pays $2.00, which we estimate to be approximately $15.00 per hour as well. We collect a total of 4,340 questions, with 620 in each of the seven sources, further divided into 310 each for the standard and adversarial methods. Each passage is paired with only one question. We randomly sample two out of five validation votes to validate the collected examples and use the remaining three votes for measuring human performance. In the validation, we regard a question as valid if at least one of the two votes is the same as the writer's gold answer. If both votes are the same as the gold answer, the question is regarded as a high-agreement example. We find that 90.3% of the collected questions are valid (92.0% for standard collection and 88.7% for adversarial collection). In addition, 65.7% of the collected questions are classified as high-agreement (68.7% and 62.7% for standard and adversarial collection, respectively). We present the dataset and worker statistics in Appendices B and C. 6954 All valid examples High-agreement portion Source Method Human UniQA DeBERTa M-Avg. Table 1 displays human and model performance. We use the questions that are validated using two out of five human votes in the validation step above and take the majority vote of the remaining three votes to measure human performance on them. We observe 3.3% and 2.0% gaps between the standard and adversarial collection in the valid and high-agreement questions, respectively. To establish the model performance that is not biased towards a single model, we compute the average accuracy ( M-avg. ) of eight different models from the following two classes: RoBERTa large (four models with different random seeds; Liu et al., 2019) and DeBERTa large and xlarge (v2; He et al., 2021) either fine-tuned on MNLI (Williams et al., 2018) first or not. The RoBERTa and DeBERTa models are all fine-tuned on RACE. Among these models, DeBERTa xlarge (MNLI-fine-tuned) performs best on RACE, achieving 86.8% accuracy. Because UnifiedQA 3B (72.3% on RACE) is used in the adversarial data collection, it shows lower accuracy on the adversarial questions (not included in the average). The performance of these two models is shown for comparison in Table 1. Except where noted, we do not train the models on any collected questions. Supervised Performance For each dataset, we evaluate the performance of DeBERTa large trained on the datasets other than the target dataset in a leave-one-out manner. Our motivation is to see whether the accuracy values significantly improve by training (i.e., the humanmodel gaps decrease). If there is a large gain, it would imply that the datasets have simple patterns among examples that the models can exploit. The results show no significant gains in the adversarial datasets, but the standard datasets show some small gains (Appendix D). and Lipton (2018) point out, reading comprehension datasets might have annotation artifacts that enable models to answer questions without passages or question sentences. To investigate such artifacts in our collected examples, we evaluate the performance of two DeBERTa models (xlarge and large fine-tuned on MNLI), which are stronger than the others, with the ablation of questions ( P+A ), passages ( Q+A ), and both questions and passages ( A only ). We see large drops in the zero-shot performance of DeBERTa xlarge. In addition, we do not observe a significant performance improvement in the supervised performance by DeBERTa large (MNLI-fine-tuned). These results demonstrate that the collected questions and answer options do not have severe annotation artifacts for any passage source (Appendix E). Following Nangia et al. (2021), we compute the humanmodel performance gap (  ) between the human and the average model accuracies to estimate the difficulty of questions for models. We observe a small variation in the gap for different passage sources in the high-agreement questions ( = 14 . 9  3 . 6) . We find the highest human performance for MCTest questions in the high-agreement portion and the lowest for Gutenberg, whereas the model's highest performance is for Slate and the lowest for MCTest. Surprisingly, the questions sourced from MCTest, which consists of simple narrative passages, show the largest gap out of all sources for the high-agreement questions. Although ReClor consists of passages for graduate-level exams, it produces smaller gaps than RACE, which consists of passages for middleand high-school English exams. Gutenberg passages are written for adults, but the examples written for those passages do not show larger gaps than those for MCTest passages. We find a trend in the human performance: the questions of easy-to-read sources (e.g., MCTest and RACE) show higher accuracy and those of difficult-to-read sources (e.g., Gutenberg and Slate) show lower, but this trend is not observed either in the machine performance or humanmachine performance gap. These observations are inconsistent with our initial expectations in the introduction. We analyze how the linguistic aspects of the collected examples correlate with the humanmodel performance gap computed in the experiments. To get a better estimate of human performance, we use the high-agreement examples (Nie et al., 2020). For ease of comparison, we split these examples into two subsets: easy (   20% ) and hard (   40% ). These subsets have 1,970 and 547 examples, respectively. Appendix F provides the frequency of easy and hard examples across the passage sources and collection methods. We compute the correlation between the human model performance gap and readability measures across all valid examples (Pearson's r and p -value) and independence between the distributions of the easy and hard subsets about the measures ( p -value in Welch's t-test). Figure 2 shows the density distributions of the easy and hard subsets, while Appendices G to L provide the plots of all valid examples. left in Figure 2). Across all examples, we observe r = 0 . 01 ( p = 0 . 47 ) (the full plot is in Appendix G). The t-test shows p = 0 . 51 . We observe no relationship between the passage length and question difficulty. We also analyze question and option length in Appendix H. FleschKincaid Grade Level We use the FleschKincaid grade level (Kincaid et al., 1975) as a basic metric of text readability (top center in Figure 2). This metric defines readability based on an approximate US grade level with no upper bound (higher is more difficult to read). It is computed for a passage using the average number of words that appear in a sentence and the average number of syllables in a word (Appendix I). The correlation between the grade and humanmodel performance gap is r =  0 . 08 ( p < 0 . 001 ) and the t-test shows p < 0 . 001 . This result demonstrates that passage readability has a small negative effect on the question difficulty, perhaps pointing to an interfering effect whereby our pre-qualified human annotators are more likely to make mistakes on more complex passages. Syntactic and Lexical Surprisal The Flesch Kincaid grade level only considers sentence length and the number of syllables. To better estimate the passage difficulty in terms of the psycholinguistic modeling of human text processing, we use syntactic and lexical surprisal measures (Roark et al., 2009). These measures are computed using incremental parsing and proved to be useful for predicting human reading time. We observe r = 0 . 000 ( p = 0 . 99 ) for syntactic surprisal and r =  0 . 007 ( p = 0 . 66 ) for lexical surprisal across all examples. We do not observe any statistically significant difference between the easy and hard subsets (syn-tactic p = 0 . 52 and lexical p = 0 . 57 in the t-test; see top right in Figure 2). Appendix J describes details of the calculation. Annotation Speed Inspired by the psycholinguistic study of text complexity (Gibson, 1998; Lapata, 2006), we measure the average time crowdworkers spent answering questions in the validation tasks (see bottom left in Figure 2). This measures the elapsed time of both reading a given passage and thinking about its question, which is used as an approximation of reading time (as a proxy of text readability). The correlation coefficient ( r =  0 . 06 with p < 0 . 001 ) and t-test ( p = 0 . 88 ) show that there is only a small negative correla-w h a t w h i c h w h y h o w w h o w h e r e i s o t h e r s i s w a s d i d d o e s c a n ... o f ... d i d w a s d o e s i s w o u l d a r e d o ... d i d d o e s m a n y w a s d o i s ... i s ... d i d ... r e a s o n ... m a i n ... n o t ... m i c h a e l ... a u t h o r ... b e w e ... ... f o ll o w i n g ... ... n o t ... j o hn ... a u t h o r ... i t ... y o u n o t p e o p l e ... n i t r o u s ... p e o p l e s c i e n t i s t s ... ... j a m e s ... a u t h o r ... y e a r s ... o l d ... s h a r i t a ... r e a li s m ... ... k r i s hn a b a rr y m o r e n o t ... ... m a d a m e ... ... t h a t ... ... tion with question difficulty. We also measure the elapsed time for writing questions as a reference (bottom center in Figure 2 and Appendix K), observing that there is no strong correlation ( r = 0 . 02 with p = 0 . 27 ). Word Frequencies Following Chen and Meur-ers (2016), we analyze the effect of word frequencies on text readability. Using word frequencies per one million words in SUBTLEXus (Brysbaert and New, 2009), we calculate the average frequency of words appearing in a passage as a measure of passage difficulty in terms of vocabulary (a lower average frequency implies greater difficult). We do not observe any statistically significant difference by the t-test p = 0 . 14 (bottom right in Figure 2) or Pearson's r = 0 . 02 with p = 0 . 27 (Appendix L). We observe similar trends even when using the human performance as the difficulty measure (Ap-pendix N). We analyze how passage sources and collection methods affect question types in this section. Question Words We automatically extract the first wh -words that appear in each valid question; if no wh -word is extracted, we count the question as polar. Figure 3 plots the question words and their two subsequent words (except articles) in the easy and hard questions. From this we observe that the hard questions are generic, not specific to given passages (e.g., which of the following is correct? ) more often than the easy questions. This probably results from the difference between the standard and adversarial data collection. The workers in the adversarial collection tend to write generic questions, while those in the standard collection write questions that are more balanced (e.g., there are more easy why and how questions). We also notice 6957 factuality factoid non-factoid gestalt/attitude numeric spatial/temporal logical Comprehension types 0 5 10 15 20 25 30 35 40 F r e q u e n c y ( % ) Dir-easy Dir-hard Adv-easy Adv-hard Figure 4: Frequency of comprehension types in the easy and hard examples for each collection method. factuality factoid non-factoid gestalt/attitude numeric spatial/temporal logical that the hard subset has more how many questions. This is likely due to the fact that it is easy for annotators to learn that numeric questions often fool the adversary model. These observations imply that adversarial data collection tends to concentrate the distribution of questions towards a few specific question types (e.g., generic and numeric). This is consistent with the observations in Kaushik et al. (2021). See Appendix M for details. Comprehension Types Following Bartolo et al. (2020) and Williams et al. (2020), we analyze what kind of comprehension is required to answer the collected questions. We sample a total of 980 high-agreement questions, 70 from each passage source and collection method, and then manually annotate them with one or more labels of seven comprehension types. The definitions of these types, examples, and detailed results are presented in Appendix M. Figure 4 shows the frequency of comprehension types for different question difficulties (676 easy, 172 hard) and the collection methods. We find that 868 questions have one label, 110 have two labels, and two have three labels. We can see that numeric , spatial/temporal , and logical questions appear more often in the hard subset in both collection methods. 2 Looking at the frequency across the passage sources in Figure 5, we find that there are some trends between the sources and comprehension types as follows:  Technical documents, such as those used in graduate-school-level reading comprehension exams, tend to yield logical reasoning questions (e.g., ReClor and Slate). Child-level texts tend to yield numerical reasoning questions in the standard setting (e.g., MCTest and RACE). In the adversarial setting, passages containing many numerical values tend to yield such questions (e.g., MCTest and Wikipedia arts). To collect gestalt questions or those considering the author's attitude in a given passage, passages covering subjective or argumentative topics (e.g., Gutenberg, Slate, and ReClor) are suitable. In contrast, expository passages such as Wikipedia articles are not. Narratives and related texts (e.g., MCTest, Gutenberg, and part of RACE) involve events with characters, which tend to yield spa-tial/temporal reasoning questions. Although the definitions of our comprehension types are coarse and these trends do not ensure that specific kinds of passages always yield the target comprehension type, considering passage sources might be an effective strategy for collecting questions of an intended comprehension type. Adversarial data collection for this purpose might not be useful because it may encourage workers to focus on writing only a few specific types of questions (e.g., numeric). To make an NLU benchmark useful, it has to consist of examples that are linguistically diverse and difficult enough to discriminate among state-of-the-art models. We crowdsource multiple-choice reading comprehension questions for passages extracted from seven different sources and analyze the effects of passage source on question difficulty and diversity. Although we expect that the difficulty of a passage affects the difficulty of questions about that passage, the collected questions do not show any strong correlation between the humanmachine performance gap and passage source, length, or readability measures. Our manual annotation of comprehension types reveals that questions requiring numerical or logical reasoning are relatively difficult. We also find several trends between passage sources and comprehension types. These results suggest that when creating a new benchmark dataset, we need to select passage sources carefully, so that the resulting dataset contains questions that require an understanding of the linguistic phenomena that we are interested in. This is especially important in the adversarial setting because it could concentrate the distribution of questions towards a few specific question types. We aim to accelerate scientific progress on robust general question answering, which could translate downstream to useful tools. We are not looking at possible sources of social bias, although this issue should be highly relevant to those considering sources to use as training data for applied systems (Li et al., 2020; Parrish et al., 2022). We are using Amazon Mechanical Turk despite its history of sometimes treating workers unfairly (Kummerfeld, 2021), especially in recourse for unfair rejections. We make sure that our own pay and rejection policies are comparable to in-person employment, but acknowledge that our study could encourage others to use Mechanical Turk, and that they might not be so careful. This work passed review or is exempt from the oversight of the internal review boards of the authors' institutes. We thank Saranya Venkatraman and Ethan Perez for their feedback on early drafts of this paper. For his early contributions to this project, we thank Harsh Trivedi. SS was supported by JST PRESTO Grant No. JPMJPR20C4. This project has ben-efited from financial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), Samsung Research (un-der the project Improving Deep Learning using Latent Structure ), and Apple. This material is based upon work supported by the National Science Foundation under Grant Nos. 1922658 and 2046556. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
abstain method objective result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain abstain method method objective method abstain result result result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain abstain abstain abstain method result objective objective abstain method abstain method objective method abstain method method abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method result result result objective abstain abstain method method method abstain other other other other other other other