Abstract Toxic language detection systems often falsely flag text that contains minority group mentions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic language. To help mitigate these issues, we create TOXIGEN , a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups. We develop a demonstration-based prompting framework and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model (Brown et al., 2020). Controlling machine generation in this way allows TOXIGEN to cover implicitly toxic text at a larger scale, and about more demographic groups, than previous resources of human-written text. We conduct a human evaluation on a challenging subset of TOXIGEN and find that annotators struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data improves its performance on human -written data substantially. We also demonstrate that TOXIGEN can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset. Toxic language detectors often over-rely on minority identity mentions 1 when flagging a statement as toxic, without considering the deeper semantic meaning of the statement (Dixon et al., 2018; Rttger et al., 2021). This can lead to severe under-detection of subtle hate (e.g.,  They have been bred 1 In this work, we use minority to refer to social and demographic groups that are frequently the targets of oppression, discrimination, or prejudice (RWJF, 2017), from a U.S. socio-cultural perspective. to be good at sports and entertainment, but not much else ; Figure 1) and over-detection of benign statements (e.g.,  child abuse is wrong, racism is wrong, sexism is wrong ; Figure 1). Importantly, such biases in toxicity detection risk further marginalizing or censoring minority groups (Yasin, 2018; Sap et al., 2019; Dias Oliva et al., 2020; Are, 2020; Daz and Hecht-Felella, 2021). We introduce TOXIGEN , a large-scale machine-generated dataset of 274,186 toxic and benign statements. To create this dataset, we leverage the massive pretrained language model GPT-3 (Brown et al., 2020), which is known to produce close-to-human-like text (Clark et al., 2021; Dou et al., 2021) but also easily generates socially biased and toxic content (Sheng et al., 2019; Gehman et al., 2020). While such human-like bias and toxicity poses real threats, we use this undesirable behavior in models like GPT-3 to improve existing toxic language classifiers, providing a path forward for mitigating systemic bias. Created using demonstration-based prompting and pretrained toxicity classifiers, TOXIGEN covers over 135k toxic and 135k benign statements about 13 minority identity groups (e.g., African Americans, women, LGBTQ+ folks, etc.). Using this machine generated approach has two advantages over scraping posts from the web as done by previous work (e.g., Davidson et al., 2017; Founta et al., 2018; Zampieri et al., 2019). First, it allows us to limit spurious identity-toxicity correlations (Dixon et al., 2018; Zhou et al., 2021) by generating equal numbers of toxic/benign statements for each demographic group, including those that are often overlooked in toxic language corpora (e.g., Native Americans). Second, machine generation and careful prompting enables us to generate implicit toxicity (i.e., without swearwords or slurs), which is by definition hard to detect or find and thus often missing in toxic language corpora (Wiegand et al., 2021). Indeed, 98.2% of TOXIGEN statements are implicit , i.e. , devoid of explicit profanity, 3309 generator classifiers ALICE Hate Scores Statements You can tell if a woman is a good person or not by how much she covers her skin 10% hate If you have ever been to a restaurant, you have probably noticed that the service is much better if the waiter is white, and the food is much better if the chef is white 8% toxic They have been bred to be good at sports and entertainment, but not much else. To generate a challenging subset of TOXIGEN , we introduce ALICE , 2 an adversarial classifier-in-the-loop decoding algorithm. We use ALICE to control the toxicity of output text by pitting a toxicity classifier against a text generator during beam search decoding. Given a toxic prompt, we can encourage generations to be less toxic based on the classifier scores. Similarly, we can steer a language model with neutral prompting towards higher toxicity generations. Our experiments with five publicly-available toxicity classifiers show that the generated sentences in both cases above fool toxicity classifiers (see Figure 1). We validate the quality of our machine-generated dataset through a comprehensive human evaluation. Our results show that on a sample of 792 machine-generated sentences, 90% could be mistaken for human-written text. We also find that the generated data indeed contains a wide variety of specific references to the minority groups mentioned in the prompts (4.2). This indicates that our data generation approaches (with or without ALICE ) successfully control the generation towards the desired toxicity and minority group mention. E xemplars 4 Delphi does not produce toxicity probabilities, so we use Open AI's content filter to game Delphi. A Delphi author has confirmed probabilities will be available soon. fine-tuning existing classifiers on TOXIGEN consistently improves performance (+719%) on 3 existing human -written implicit toxic datasets: ImplicitHateCorpus (ElSherief et al., 2021), SocialBiasFrames (Sap et al., 2020), and DynaHate (Vidgen et al., 2021). This indicates that the dataset generated in this work and the approaches for generating data provide major steps towards improving toxicity classifiers, and could potentially be used downstream to address the issues from biased machine generation (Sheng et al., 2019) or neutral toxic degeneration (Gehman et al., 2020). Detecting implicit toxicity about minority groups (e.g., stereotyping, microaggressions), remains an elusive goal for NLP systems (Han and Tsvetkov, 2020; Wiegand et al., 2021). One key challenge is that, in contrast to explicit toxicity, implicit toxicity is not marked by the use of profanity or swearwords, is sometimes positive in sentiment, and is generally harder to detect or collect at scale (MacA-vaney et al., 2019; Breitfeller et al., 2019). Nonetheless, implicitly toxic language about minority or marginalized groups is often psychologically damaging to members of those groups (Sue et al., 2007; 3 https://github.com/microsoft/ToxiGen 3310 Datasets Properties Source Size % Implicit % Hate Class Breitfeller et al. (2019) Reddit 2,934 99.4 100.0 TweetBLM (Kumar and Pranesh, 2021) Twitter 9,165 99.0 33.7 de Gibert et al. (2018) StormFront 9,916 92.2 11.3 Waseem (2016) Twitter 16,914 82.4 31.7 ImplicitHateCorpus (ElSherief et al., 2021) Twitter 22,584 96.8 39.6 Davidson et al. (2017) Twitter 24,802 30.2 5.0 Kennedy et al. (2018) Hate Forums 27,665 71.8 9.1 DynaHate (Vidgen et al., 2021) Human-Machine Adv. 41,134 83.3 53.9 SocialBiasFrames (Sap et al., 2020) Social Media 44,671 71.5 44.8 Founta et al. (2018) Twitter 80,000 26.1 7.5 TOXIGEN (ours) GPT-3 274,186 98.2 50.1 Table 1: Comparing toxic language datasets. Nadal et al., 2014; Kanter et al., 2017; Nadal, 2018; Saleem and Anderson, 2013) and can reinforce stereotypical or hateful perceptions of them (Behm-Morawitz and Mastro, 2008; Soral et al., 2018). A second challenge for detecting subtle toxicity about minority groups is that minority mentions are more often the targets of social biases and toxicity (Hudson, 2017). As such, minority mentions often co-occur with toxicity labels in datasets scraped from online platforms (Dixon et al., 2018). For example, over 93% of mentions of Jewish folk in Sap et al. (2020) are toxic (Wiegand et al., 2021). In turn, models trained on such data can exploit these spurious minority-toxicity correlations instead of considering the deeper semantics of text (Zhou et al., 2021). Importantly, the spurious correlations are also learned by large language models, which are known to produce stereotypical, biased, or toxic content when prompted with minority mentions (Sheng et al., 2019). Given that the main mitigation approach to prevent Large Language Models (LLM) from generating toxic language is to train new classifiers to detect such language, these classifiers also learn the spurious correlations and start blocking most language referencing minority groups. This risks erasure (Xu et al., 2021). With TOXIGEN , we aim for generating a large scale dataset that represent implicit toxicity while balancing between toxic and benign statements, to address the gaps of previous work. As shown in Table 1, existing datasets contain large amounts of explicit toxicity. While valuable, most previous work has relied on scraping data from online platforms, which leads to dataset imbalances with respect to minority-mentioning posts that are toxic vs. benign. Examples are collected at scale using keyword-based scraping approaches (Waseem, 2016; Davidson et al., 2017; Zampieri et al., 2019), the bootstrapped scraping approaches (Founta et al., 2018), and machine-vs-human adversarial data collection (Dinan et al., 2019; Vidgen et al., 2021), among others. In contrast, using large language models to generate our dataset allows us to control the minority groups mentioned in our statements, as well as their implicitness, at larger scale. To create TOXIGEN , we use demonstration-based prompting for LLMs, encouraging a text generator to produce both toxic and benign sentences that mention minority groups without using explicit language. We introduce a classifier-in-the-loop decoding method based on constrained beam search, ALICE , which, along with samples generated without ALICE , contributes to generating a challenging subset of TOXIGEN . Using these methods, we generate a massive set of statements (over 274,000) containing equal numbers of toxic and benign sentences for 13 identity groupssee Table 2. TOXIGEN is generated by prompting a language model to produce both benign and toxic sentences that (1) include mentions of minority groups by name and (2) contain mainly implicit language, which does not include profanity or slurs. To achieve this, we perform demonstration-based prompt engineering: Acquiring example sentences, 3311 the only way to civilize the natives... passing them to a LLM, 4 and collecting the subsequent response. For example, when shown ten sentences mentioning Muslim folks neutrally , GPT-3 often responds with a similar neutral sentence. Prompts are text fragments passed into language models that can encourage certain behaviors (Brown et al., 2020). However, designing prompts is notoriously challenging (Liu et al., 2021c). While there are several approaches for prompting pretrained LLMs (Liu et al., 2021b), a recent and promising direction is demonstration-based prompting (Gao et al., 2021; Mishra et al., 2021). Here, example statements are passed to an LLMs, encouraging it to produce a similar, but distinct, statement. For example, to encourage LLMs to produce benign statements about Asian people, we might use the following examples: (1) many impactful people are of asian descent , (2) there are hundreds of asian cultures . To engineer prompts that lead to high-quality, group-mentioning toxic and benign statements at scale, we first gather and curate sets of examples. lect many examples. Intuitively, given many examples of benign sentences that mention one particular group, a language model can be used to produce more. For benign prompts, we encourage realistic text generation and include diverse voices by collecting benign sentences from blog posts and news articles that mention a group. However, finding large amounts of such data at scale is challenging this is why implicit datasets are hard to acquire. To build a large enough set of demonstrations, we begin with a small number of examples from the wild, then engage a human-in-the-loop process: collect some demonstrations, pass them to our LLM, comb through many responses, and add the best examples to a growing set. Ensuring that a set of examples consistently produces benign responses that still mention the targeted minority group is challenging and so we iterate this loop many times, sampling random subsets of our examples to serve as prompts and observing the responses. This way, we collect 20-50 demonstration sentences per group, all of which we release. To encourage implicit toxicity from a LLM, we find examples of human-written sentences with implicit toxicity towards each group from hate forums (de Gibert et al., 2018) and Reddit (Breitfeller et al., 2019). We repeat the human-in-the-loop process to expand our sets of examples. Overall, by repeating this process for both toxic and benign examples for all 13 target groups, we create 26 sets of prompts, 3312 Group Count Avg. Demonstration-based prompting alone consistently produces toxic and benign statements about minority groups (see Section 4). There is no guarantee that these statements will be challenging to existing toxicity detectors. Therefore, we also develop ALICE , a variant of constrained beam search (CBS; Anderson et al., 2017; Hokamp and Liu, 2017; Holtzman et al., 2018; Lu et al., 2021) during decoding that generates statements that are adversarial to a given pre-trained toxicity classifier. ALICE creates an adversarial game between a pre-trained language model (PLM) and a toxicity classifier (CLF) during constrained beam search decoding. In many CBS settings, constraints are added during beam search decoding to force the model to either include or exclude a specific word or group of words in the output (Anderson et al., 2017; Hokamp and Liu, 2017; Lu et al., 2021). With ALICE , we instead want to enforce soft constraints on the probabilities coming from a given toxicity classifier CLF during beam search: 5 log p ( w i +1 | w 0: i )   L log p LM ( w i +1 | w 0: i ) +  C log p CLF ( w 0: i +1 ) (1) Here,  L and  C denote hyperparameters that determine the respective contribution of the language model and classifier to the decoding scoring function. By using this weighted combination, we can steer generations towards a higher or lower probability of toxicity without sacrificing coherence enforced by the language model. To create examples that challenge existing toxicity classifiers, we use two adversarial setups:  False negatives : We use toxic prompts to encourage the language model to generate toxic outputs, then maximize the classifier's probability of the benign class during beam search. False positives : We use benign prompts to encourage the language model to generate nontoxic outputs, then maximize the probability of the toxic class during beam search. In the first approach, we are also able to detoxify model outputs when the classifier successfully steers the generations towards non-toxic language. ALICE is illustrated in Figure 2. We generate TOXIGEN data with and without ALICE. Without ALICE, we use topk decoding (Fan et al., 2018) alone with our toxic and benign prompts. With ALICE, we use the HateBERT fine-tuned OffensEval model from Caselli et al. (2021) as the toxicity classifier (CLF). This model covers a range of direct and veiled offense types. We use GPT-3 for the language model. For decoding, we use  L =  C = 0 . 5 , a maximum generation length of 30 tokens, a beam size of 10, and a temperature of 0.9. Due to limitations imposed by the OpenAI GPT-3 API on accessing log probabilities for the full model vocabulary, we restricted the vocabulary 5 This is similar in spirit to previous work on using cooperative discriminators on uncontrolled LLMs (Holtzman et al., 2018; Krause et al., 2020; Yang and Klein, 2021; Liu et al., 2021a), yet in this work our LLM is controlled in an adversarial way by prompting and by a classifier. size to the top 100 tokens, and then resample from the allowed tokens (tokens not appearing in the prompt) using topk . 6 3.4 TOXIGEN Statistics Statistics of TOXIGEN are presented in Table 2. In our final dataset, generation length varies significantly and, as expected, almost all the statements are implicit. As we show in 4, the ALICE generated data is successful at attacking the given toxicity classifier, contributing a challenging, adversarial subset of TOXIGEN . 7 In the released data, we split off a test set that is validated by human annotators (see 4.2). To ensure the quality of TOXIGEN , we conduct human validation experiments and create TOXIGENHUMANVAL , a human-validated test set. Specifi-cally, we investigate the reliability of our prompt-based and ALICE -based methods at generating human-like statements and controlling statements' toxicity and the minority groups mentioned (4.2). Additionally, we measure the effectiveness of ALICE -generated statements (vs. topk -generated) at fooling classifiers (4.3). For each generated statement, we ask the annotators various questions, described below, that take into account multiple dimensions of how toxic 6 We force beam search decoding to not use tokens from the prompt to prevent direct copying. Certain tokens appearing in the prompt such as punctuation are allowed. 7 We compute the % of implicit samples using https: //github.com/RobertJGabriel/Google-profanity-words , the same as ElSherief et al. (2021), also removing ambiguous terms (e.g., bloody). machine-generated language presents a potential harm to readers. See Appendix B for an annotation screenshot and other study details. Perceived hatefulness with respect to human or AI-authored text. We first ask annotators to guess whether the statement's author was a human or an AI system (HUMANORAI). Then, we ask whether the statement would be harmful to anyone if an AI system wrote it ( HARMFULIFAI), as well as if a human wrote it ( HARMFULIFHUMAN ); we hypothesize that readers may have different standards for machine-generated text than human-written text. For all questions measuring harmfulness of text, we consider potential harm on a 1-5 scale with 1 being clearly benign and 5 indicating very offensive or abusive text. Perceived intent of the writer. We ask readers whether statements were likely intended to be harmful ( HARMFULINTENT ), since some biased statements can be positively intended (e.g., benevolent sexism; Glick and Fiske, 1996). Additionally, we ask if the statement exhibits a positive stereotype (POSSTEREO ), which is also harmful (e.g., model minority myths; Cheryan and Bodenhausen, 2000). Detailed harm explanations. To better understand how harm may be perpetrated against the minority group, we ask readers in-depth questions about text's content, following Sap et al. (2020) and Olteanu et al. (2018). We ask whether or not the statement is lewd or sexual (LEWD ), whether and how it references the targeted group or other groups ( WHICHGROUP , GROUPFRAMING ), whether it claims to be factual or opinion (FACTOROPINION ). Data and Setup. We selected 792 statements from TOXIGEN to include in our test set, such that no training statement had cosine similarity above 0.7 with any test statement. Each test statement was then rated by 3 annotators from a pool of 156 prequalified annotators from Amazon MTurk (See Appendix B for details). Inter-annotator agreement. To investigate the quality of our annotations, we compute agreement on toxicity ratings. 8 We find that annotators agreed moderately and are higher than or equal rates to prior work on hate speech annotation (Ross et al., 8 Specifically, we take the max of the HARMFULIFAI and HARMFULIFHUMAN scores and map it into three classes (scores < 3: non-toxic, =3: ambiguous, > 3: toxic). 2017; Sap et al., 2020), with a Fleiss'  =0.46 (Fleiss, 1971) and Krippendorff's  =0.64 (Krippen-dorff, 1980). In 55.17% of cases, all 3 annotators agree, while a majority (  2/3) agree for 93.4%. Human validation results. First, we find that our machine-generated statements are largely indistinguishable from human-written statements. For examplesee Table 3human annotators often AI s p eake r H u m a n s p eake r Figure 5: Avg. predict that our text is generated by a human. In fact, on average 90.5% of machine-generated examples are thought to be human-written by a majority of annotators, as shown in Figure 4. We also note that harmful text confuses readers slightly more than non-harmful text: 92.9% of toxic examples are mislabeled as human-written compared to 90.2% for non-toxic. Most toxic examples are also hate speech (94.56%). While opinions are common in both toxic and non-toxic examples, most fact-claiming text is non-toxic. Second, we find that demonstration-based prompting reliably generates toxic and benign statements about minority groups (4.3). Further, for the machine-generated examples, we find that 30.2% are harmful (given a score of > 3), while only 4% are ambiguous. This indicates that these data are sufficiently toxic or benign. We also find 3315 that all identity groups covered by the dataset were represented in the human study (see Figure 3), and observe that the identity group referenced by the prompt is generally the same as the group referenced by the corresponding TOXIGEN text, though there is some deviation. This is likely due to GPT-3 conflating identities or mentioning multiple groups. Interestingly, there is no significant difference in toxicity when we account for whether annotators perceive scores as written by humans or AI (Figure 5). This finding indicates that our machine-generated text is perceived as similarly harmful to human text. We also find that the most common framing tactic is moral judgement, or questioning the morality of an identity group, which has been linked to toxicity by prior work (Hoover et al., 2019). As further validation, we investigate whether ALICE -generated statements are more adversarial compared to topk -generated ones. For 125 randomly-selected prompts (62 toxic and 63 non-toxic), we generate two statements: one with ALICE and one without (topk ). We then collect annotations for the 250 statements using the setup described in 4.1, and get toxicity scores from HateBERT. We find that for topk sampled sentences, the prompt label indeed matches the desired label (95.2% of non-toxic examples and 67.7% of toxic examples). For ALICE, 40.3% of toxic examples match the prompt label and 92.1% of non-toxic examples match. We also find that ALICE succeeds in fooling HateBERT (26.4% of ALICE -decoded sentences fool HateBERT vs. 16.8% of topk sampled sentences). Finally, ALICE is effective for detoxifying generated text: the avg. human-annotated toxicity score for ALICE -decoded sentences with a toxic prompt is 2.97, compared to 3.75 for top-k . This difference is statistically significant with p < 0 . 001 . ALICE therefore leads to harder, more ambiguous examples. We greatly expand on these findings in Appendix E with a larger scale human evaluation (  10,000 samples) comparing sentences generated with and without ALICE . To further showcase the usefulness of TOXIGEN we investigate how it can enhance classifiers' abilities to detect human-written and machine-generated implicit toxic language. We fine-tune Test Data Finetune Data None ALICE topk ALICE + topk H a t e BERTSBF test 0.60 0.66 0.65 0.71 IHC 0.60 0.60 0.61 0.67 DYNAHATE 0.47 0.54 0.59 0.66 TOXIGEN-VAL 0.57 0.93 0.88 0.96 R o BERT a SBF test 0.65 0.70 0.67 0.70 IHC 0.57 0.64 0.63 0.66 DYNAHATE 0.49 0.51 0.50 0.54 TOXIGEN-VAL 0.57 0.87 0.85 0.93 Table 4: AUC for HateBert and RoBERTa both zero-shot and fine-tuned on 3 versions of our dataset: ALICE only, topk only, and both combined. the widely-used HateBERT (Caselli et al., 2021) and ToxDectRoBERTa (Zhou et al., 2021) models on the training portion of TOXIGEN , using the prompt labels as proxies for a true toxicity label. Then, we compare the performance of the out-of-the-box models to those fine-tuned on TOXIGEN on three publicly available human-written datasets (IMPLICITHATECORPUS (ElSherief et al., 2021), the SOCIALBIASFRAMES test set (Sap et al., 2020), and DYNAHATE (Vidgen et al., 2021)) as well as the evaluation portion of our machine-generated dataset (TOXIGEN-HUMANVAL ). To ablate the contribution of each decoding method, we also split TOXIGEN into equal numbers of ALICE-generated and topk -generated examples. Our resultssee Table 4show that fine-tuning HateBERT and ToxDectRoBERTa on TOXIGEN improves performance across all datasets. The improvement on human-written datasets shows that TOXIGEN can be used to improve existing classifiers, helping them better tackle the challenging human-generated implicit toxicity detection task. Fine-tuned HateBERT performs strongly on TOXIGEN-HUMANVAL , demonstrating that our data can successfully help guard against machine-generated toxicity. In this work, we used a large language model to create and release TOXIGEN , a large-scale, balanced, and implicit toxic language dataset. TOXIGEN is 3316 far larger than previous datasets, containing over 274k sentences, and is more diverse, including mentions of 13 minority groups at scale. The generated samples are balanced in terms of number of benign and toxic samples for each group. We proposed ALICE , an adversarial decoding scheme to evaluate robustness of toxicity classifiers and generate sentences to attack them, and showed the effectiveness of ALICE on a number of publicly-available toxicity detection systems. In our experiments, we showed that fine-tuning pre-trained hate classifiers on TOXIGEN can improve their performance on three popular human -generated toxicity datasets. We also conducted a human study on a subset of TOXIGEN , verifying that our generation methods successfully create challenging statements that annotators struggle to distinguish from human-written text: 90.5% of machine-generated examples were thought to be human-written. Risks in dataset release While the purpose of our work is to curate diverse and effective hate speech detection resources, our methods encourage a large language model to make its generation more toxic. This poses a potential misuse case where bad actors exploit these methods for nefarious purposes like spreading machine-generated hate speech. Still, ignoring this possibility does not make it go away and our work introduces an opportunity for the community to push back against harm towards minority groups. Our ultimate aim is to shift power dynamics to targets of oppression. Therefore, we do not consider identity dimensions that are historically the agents of oppression (e.g., whiteness, heterosexuality, able-bodied-ness). Please also note that there is still a lot that this dataset is not capturing about toxic language. Our annotations might not capture the full complexity of these issues related to human experiences. There is need for multi-disciplinary work to better understand these aspects. ALICE The proposed method in this work attacks content filters via an adversarial game between two AI systems and thus passes the existing content filtersas we show for 5 publicly-available systems. It is important to leverage this and similar approaches to improve content filters and prevent large scale attacks against sensitive platforms. Improving Toxicity Detection Effective classifiers for machine biases are required to combat the scale of online harm. Without such systems, minority groups are likely to be targeted by current (biased) systems. Our work is a significant step towards advancing this crucial classification task. Still, toxicity is inherently subjective (Sap et al., 2021). Therefore, moving beyond binary detection tasks to a focus on more nuanced labeling systems (ElSherief et al., 2021; Leonardelli et al., 2021) will prove crucial in developing responsible systems. Relationship to Policy The topic of detecting and mitigating toxicity is relevant to the ongoing work and discussions in the space of policy and legislation for AI technology (Wischmeyer and Rademacher, 2020; Reich et al., 2021). Carefully crafted policy and regulation can play an important role in providing oversight into the development and deployment of content moderation systems and toxicity detection algorithms in practice (Benesch, 2020; Gillespie et al., 2020). Getting this right carries a crucial importance for the society as errors in content moderation can disproportionately affect minority groups (Sap et al., 2019). We see a path forward in which tools and techniques like those presented in this work are paired with human expertise and well-informed policy & regulation in bringing scalable and reliable solutions to practice. We acknowledge and encourage the critical role the NLP research community is poised to play in this inter-disciplinary effort. We thank Azure AI Platform and Misha Bilenko for sponsoring this work and providing compute resources, Microsoft Research for supporting our large scale human study, and Alexandra Olteanu for her feedback on human evaluation. We also thank the crowdworkers for their time and effort.
abstain abstain objective abstain abstain result result result objective abstain method abstain abstain abstain method method result abstain abstain abstain result other method method method abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain other method method method abstain abstain method method method abstain method method abstain method abstain abstain abstain abstain abstain method abstain method method method abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method abstain method abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain abstain objective result result abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other other