While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present P olyjuice , a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that P olyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three di  erent tasks (with around 70% less annotation e  ort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts. Counterfactual reasoning  mentally simulating what would have happened if conditions were different  is a common tool for making causality assessments (Kahneman and Tversky, 1981), which in turn are crucial for model evaluation, error analysis, and explanation (Miller, 2019). For example, in Figure 1, It is great for kids is perturbed into multiple variations, each providing unique insights by simulating what would have happened if the sentence was di  erent. Applications of counterfactual reasoning to NLP generally specify the relationship x (cid:41)  x , and then create  x according to the relationship. As a result, prior work has tailored counterfactual generators for di  erent applications, only collecting subsets of  x that are useful for the specific task. For example, to support model training and evaluation , human annotators create counterfactuals It is great for kids. that change the groundtruth labels by manually rewriting instances (Gardner et al., 2020; Qin et al., 2019) or defining perturbation functions (Ribeiro et al., 2020). Manual rewrites are costly ( e.g., 45 minutes per counterfactual (Kaushik et al., 2020)) and susceptible to systematic omissions ( e.g., human annotators may cover great (cid:41) not great, but miss kids (cid:41) no one in Figure 1B). Meanwhile, automated generators for model analysis and explanation usually focus on other relationships, e.g., generating  x that have di  erent model predictions than x (Ross et al., 2020; Zhang et al., 2019a). As a result, they neglect prediction-preserving counterfactuals that are equally important for understanding or shaping model behaviors, like kids (cid:41) no one and great (cid:41) scary linked to Figure 1D. applica-1 We open source P olyjuice at https://github.com/ tongshuangwu/polyjuice . tions. Moreover, for cases like model explanation and analysis, a general-purpose pool of counterfactuals may be preferable, as the relationship of interest can be more exploratory and user-oriented (Wu et al., 2019). In this work, we formalize the task of counterfactual generation , disentangling generation from the application of counterfactuals. Given an input x (Figure 1A), our generator produces a set of counterfactuals  X = {  x 1 ,  x 2 , ... } with application-agnostic relationships x (cid:41)  x i (Figure 1B). Afterwards, we use application-specific selection methods to find subsets of  x that are most e  ective for a given use case (Figure 1C). We frame the generation step as conditional text generation, and finetune GPT-2 (Radford et al., 2019) into a generator called P olyjuice using ( x ,  x ) pairs. To allow for targeted counterfactuals, we also design control codes like negation or delete (Figure 1B), and adopt fill-in-the-blank structures (Donahue et al., 2020) to specify where the perturbation occurs and how. Intrinsic evaluation shows that P olyjuice generates  x that are fluent , diverse , and close to x , and that the control mechanisms retrieve perturbations that would likely not be sampled from o  -the-shelf language models. With simple selection heuristics, we show that a single P olyjuice model can significantly aid humans in diverse downstream applications. 2 For counterfactual training and evaluation (3), humans label P olyjuice counterfactuals rather than creating them from scratch. They produce training data that significantly improve model generalization, as well as contrast sets that help identify model vulnerabilities (Gardner et al., 2020), with around 70% less annotation e  ort. In another application, P olyjuice produces counterfactual explanations (4), providing significant insight on top of state-of-the-art explanation techniques. Finally, P olyjuice supports counterfactual error analysis (5). It allows users to explore related counterfactuals ( e.g., the model responds di  erently to di  erent negation forms in Figure 1B), and to aggregate individual counterfactuals into patterns in order to gain systematic understanding of model behavior. Given an instance x , a generator g produces a set of counterfactuals  X = {  x 1 ,  x 2 , ... } with various re-2 lationships x (cid:41)  x i . For example, great (cid:41) not great, kids (cid:41) no one in Figure 1B are both instances of the negation relationship. Each ( x ,  x ) pair shares multiple relationships  these two are also instances of the label flipping relationship if the task is sentiment analysis (but might not be for other tasks). As illustrated in 1, knowing which relationships apply aids selection for downstream applications. We expect g to produce counterfactuals  x that are (1) close to x , preferably only involving the minimal changes necessary to establish a certain effect (Pearl, 2018), allowing users to make causality assessments. The generated  x should also be (2) fluent , i.e., grammatically correct (Morris et al., 2020) and semantically meaningful ( e.g., Colorless green ideas sleep furiously is not meaningful (Chom-sky, 2002)). Fluency operationalizes probable counterfactuals in the context of NLP; as Kahneman and Tversky (1981) stated, humans strongly favor counterfactuals that are close to the original instance, but also prefer those that could have easily happened without assuming rare events or strange coincidences. Further, as a general-purpose generator, g should produce counterfactuals with a measure of (3) control over relationships x (cid:41)  x , such that the counterfactuals can vary with the object-of-attention in each application (the focus rule (Kahneman and Tversky, 1981)). Finally, we expect g to output a (4) diverse set of  x in terms of relationships, covering a large variety of what-ifs for di  erent applications (Pearl, 2018). Prompt format design. To ensure that  x is close to x rather than arbitrary text, we condition the generation on x , followed by a special token (Line 1 in Figure 2A). In Line 2, we have control codes (Keskar et al., 2019) such as negation . We design them to specify types of perturbation from among lexical, syntactic, or semantic aspects (see Table 1), inspired by prior work that categorizes manually created counterfactuals (Kaushik et al., 2020; Gardner et al., 2020). As an additional layer of control over x (cid:41)  x , we allow users to specify where changes happen by having the LM infill [BLANK] tokens (Donahue et al., 2020), rather than generating arbitrary counterfactuals (Lines 34). Finetuning GPT-2  a causal LM for predicting next tokens  additionally allows us to exercise control at various levels of granularity. At generation time, if the user provides only the original example, P olyjuice will generate the control code, the blank locations, and the infilling (Lines 24). Alternatively, the user can specify the control code, or the control code and the blanks, to exercise different degrees of control depending on the application. As later shown in 4 and 5, such control is important for di  erent use cases. Training data. To train a conditional model, we combine six existing sentence-pair datasets, each containing a subset of the desired phenomena in Table 1. Further, we find naturally occurring sentence pairs (filtered by edit distance to guarantee closeness) in non-paired datasets including CommonGen (Lin et al., 2020), Natural Questions (Kwiatkowski et al., 2019), and SQuAD (Ra-jpurkar et al., 2016), such that the resulting dataset contains diverse counterfactuals. 3 We translate these sentence pairs into the format given in Figure 2A. For each ( x ,  x ), we compute its primary control code using part-of-speech tags and dependency trees. For example, negation occurs when we observe changes to negation modifiers or specific words like supposedly, and shuffle occurs when we have overlap between tokens deleted and added. When multiple changes occur, we label it with the control code which most significantly changes the semantics of the corresponding subphrase as computed by SBERT (Reimers and Gurevych, 2019). For example, in Figure 2A, negation (great (cid:41) not great) is more significant than lexical (kids (cid:41) children). To balance the distribution (Table 7 in Appendix A), for each dataset, we extract control codes from all the ( x ,  x ), 4 and randomly sample up to 10,000 instances per codes. We exclude data related to our applications, e.g., PAWS-QQP (Zhang et al., 2019b). 4 We use sentences in a pair interchangeably as x and  x to learn the control codes both ways. tures related to the perturbed spans (Figure 2B), including (1) just the changed tokens, (2) the associated parsing structures, (3) the merged changes, and (4) the entire sentence. We eventually obtain 657 , 144 prompts from 186 , 451 pairs. Fluency filtering. While the original GPT-2 produces fluent text, some combinations of control codes and blanks cause P olyjuice to generate nonsensical results. Following Morris et al. (2020), we score both x and  x with GPT-2, and filter  x when the log-probability (on the full sentence or the perturbed chunks) decreases by more than 10 points relative to x . Fully automated uses of P olyjuice ( e.g., adversarial attacks) may benefit from stricter constraints, at the cost of diversity (as surprising changes may be filtered even if they are fluent). We evaluate P olyjuice on closeness and diversity by comparing its perturbations on 300 randomly selected sentences with baselines that use more or less context from x : (1) non-finetuned GPT-2, (2) token-infilling RoBERTa (Liu et al., 2019) and (3) span-infilling T5 (Ra  el et al., 2020). As shown in Table 2, P olyjuice generates counterfactuals that are close to the original instance, measured by syntactic tree (Zhang and Shasha, 1989) and Levenshtein edit distance (Levenshtein, 1966). In contrast, non-finetuned GPT-2 generates arbitrary text instead of perturbations when given the starting tokens of a sentence, as it only leverages context in a single direction. As for infilling models, P olyjuice counterfactuals are more diverse (measured by self-BLEU (Zhu et al., 2018)) than RoBERTa ones, which is restricted to word substitution. Meanwhile, T5 displays higher diversity but less closeness, probably due to the fact that it does not consider the original masked tokens when generating  x . For example, in Figure 1 It is great for kids, T5 replaces for kids with idea, to meet you, whereas P olyjuice generates for kids yet adults can enjoy, for any audience. We evaluate controllability by comparing P olyjuice with T5 as well as with GPT-2 finetuned on prompts without codes. We verify that the codes improve the success rate of generating counterfactuals with the desired perturbation types set out in Table 1 by as much as 42% for perturbations such as negation and insert . For example, given It is [BLANK] great for kids, baselines generate also, fun and, rather than not ( negation ). We further verify the fluency for P olyjuice counterfactuals in three tasks / datasets: (1) Sentiment Analysis, SST-2 (Socher et al., 2013), (2) Natural Language Inference ( NLI ), SNLI (Bowman et al., 2015), and (3) Duplicate Question Detection (QQP) (Wang et al., 2019). We randomly select 100 sentences per dataset, generate 3  x per x , and ask crowd workers to rate whether they are likely written by native speakers. The workers rated most counterfactuals as fluent: 78% in SST-2, 76% in QQP, and 86% in SNLI. In subsequent sections, we show these rates are suitable for applications where people team up with P olyjuice . We ask crowdworkers to label P olyjuice -generated counterfactuals for Sentiment , NLI , and QQP , for the purposes of evaluation and training. 5 In each labeling round, the worker is presented with an original x and its label, and asked to annotate the groundtruth for three  x , rejecting non-fluent ones (details and interface in Appendix B.1). We use a simple heuristic to select which counterfactuals are presented for labeling, aimed at increasing diversity. Representing each  x by its token changes, control code, and dependency tree structure, we greedily select the ones that are least similar to those already selected for labeling. This avoids redundancy in the labeling set, e.g., common perturbation patterns such as black (cid:41) white. We verify whether P olyjuice counterfactuals can be used to create contrast sets (Gardner et al., 2020), i.e., evaluation sets where each instance has a nearby counterfactual with a di  erent groundtruth, to better evaluate model decision boundaries. We 5 We collect asymmetric counterfactuals (Garg et al., 2019) by sampling more Duplicate and Entailment examples in QQP and NLI to perturb, due to the di  culty of flipping other labels. construct these sets by simply filtering out counterfactuals that are labeled the same as their original instances (40%63% depending on the task). For each task, we test multiple classifers open-sourced by Huggingface (Wolf et al., 2020), and report the best performing model for each 6 in Table 3 (results for other models are analogous). P olyjuice contrast sets display performance gaps consistent with those of Gardner et al. (2020), where the sets are constructed manually by NLP researchers, even though we use non-expert annotators who only label examples rather than creating them. Following Kaushik et al. (2020), we augment training sets with counterfactual examples. In all experiments, we finetune roberta-base on datasets of n original examples and m counterfactuals, which are generated by P olyjuice ( m-polyjuice ) or crafted from scratch by humans ( m-CAD from Kaushik et al. (2020), only available for NLI ). To distinguish the benefit of counterfactuals from that of just adding more data, we further add a baseline that uses n + m original examples ( m-baseline ). In addition to in-domain test set accuracy, we measure models' generalization on out-of-domain datasets, as well as contrast sets and challenge sets. We also evaluate model capabilities with CheckList (Ribeiro et al., 2020) for Sentiment and QQP . Reported model performances are averaged across multiple data samples and random seeds (Appendix B.2). For Sentiment , we select random P olyjuice counterfactuals regardless of their labels, as long as an original x has at least one  x that flips the label. For NLI and QQP , we observed in a pilot study that 6 huggingface.co/{roberta-large-mnli, textattack/roberta-base-SST-2,ji-xin/roberta_base-QQP-two_stage} randomly chosen counterfactuals may not be more e  ective than the same amount of additional data. We suspect that P olyjuice lacks domain knowledge and context for identifying critical perturbations, and therefore brings benefits redundant with pretraining (Longpre et al., 2020). Thus, we use the slicing functions of Chen et al. (2019) to find patterns of interest ( e.g., prepositions in NLI ), and perturb those patterns by placing [BLANK] s on the matched spans. For example, His surfboard is beneath him becomes His surfboard is [BLANK] him, and P olyjuice generates counterfactuals such as His surfboard is beneath (cid:41) next to him. Results. Tables 46 indicate that P olyjuice augmentation is e  ective in all tasks: m-polyjuice maintains in-domain accuracy while consistently improving or maintaining generalization accuracy in various out-of-domain and challenge sets. On NLI , P olyjuice counterfactuals are as e  ective or more e  ective than counterfactuals created from scratch ( m-CAD ). Notably, we obtain the largest gains on challenge and contrast sets ( e.g., Break and DNC in Table 5) or when the out-of-domain dataset is su  ciently di  erent from the training domain ( e.g., Senti140 and SemEval in Table 4). P olyjuice also improves results on CheckList tests that previously had high error rates: it significantly lowers the error rates on 11 out of 27 QQP tests, 7 making 2 / 27 tests worse. For Sentiment , it improves the model on 5 out of 15 tests, hurting 1. Here, we only report a low m / n ratio ( < 10% for NLI and QQP) to show that a small amount of augmentation is already beneficial. The results are similar for other combinations we explored (see Appendix B.2), except when the ratio of counterfactual to original data was too high ( e.g., , m = n may decrease vocabulary diversity or induce additional data bias, echoing (Khashabi et al., 2020)). We show that P olyjuice counterfactuals are useful for evaluation, and more e  ective than additional (non-counterfactual) data for training in a variety of tasks. In contrast to prior work where humans generate counterfactuals from scratch, we only ask them to label automatically generated ones, while still achieving similar or better results. We believe our approach is more e  ective than manual creation (although both are beneficial): in 7 The absolute error rate drops for at least 5 points, with a relative di  erence of more than 10%. terms of implementation e  ort, the process of just labeling counterfactuals is the same as labeling original examples, such that no additional annotator training or separate pipelines are required; in contrast, Kaushik et al. (2020) set up two separate crowdsourcing tasks for creating and labeling the counterfactuals. Further, annotator e  ort is much lower, as evaluating examples is easier than creating them  Kaushik et al. (2020) report an average of  2 minutes per NLI counterfactual prior to quality validation, while our median time was 10 seconds per counterfactual. Even after our quality validation (removing noisy annotators, disregarding non-fluent counterfactuals), our rate for NLI is  36 seconds per counterfactual (used in Table 5). In terms of the utility per counterfactual, manual creation and P olyjuice may be complementary. Manual annotation may be unreliable or incomplete for certain forms of counterfactuals (Ribeiro et al., 2018), whereas P olyjuice can miss more complex or context-dependent changes, and could benefit from target perturbations that compensate for its lack of domain knowledge (targeted guidance is also helpful for human annotators (Huang et al., 2020)). Thus, it may be important to mix both approaches (Khashabi et al., 2020). P olyjuice 's flex-ibility opens up possibilities for hybrids between human creation and human verification of targeted, machine-generated counterfactuals. Though ubiquitous, token scores may not always reflect their real importance (Pruthi et al., 2020). Popular packages like LIME or SHAP estimate scores by masking words, and therefore may not reflect model behavior on natural counterfactual cases. For example, the token friend in Figure 3A is not considered important even though a natural substitution in Figure 3B flips the prediction. The opposite happens to in depression, where a significant change makes no di  erence to the model's prediction (Figure 3C). Even perfect importance scores may be too abstract for users to gain real understanding (Miller, 2019), e.g., users may not grasp the significance of a low importance score for the token help without concrete examples such as the one in Figure 3D. Since presenting a large number of concrete counterfactuals would be overwhelming, we propose a hybrid approach, displaying feature attributions as a high-level summary, together with a judicious selection of P olyjuice counterfactuals that make behaviors concrete and highlight potential limitations. Following Miller (2019)'s observation that people look for explanations revealing unexpected behavior, we select surprising counterfactuals. 8 That is, we estimate the expected change in prediction with feature attributions, and select counterfactuals that violate these expectations, i.e., examples where the real change in prediction is large even though importance scores are low (Fig-ure 3B), and examples where the change is small but importance scores are high (Figure 3C). Of course, users can also view additional counterfactuals that perturb tokens of particular interest, a technique that we explore in the next section. User evaluation. We study the scenario where an expert has access to a model and local explanations, and evaluate the additional benefit of showing counterfactuals, i.e., whether they bring new insights. We evaluate three ways of generating counterfactuals: (1) P olyjuice -random , a baseline where we show random P olyjuice counterfactuals, (2) Expert-surprise , where two graduate students (non-participants) were given access to the model and instructed to create counterfactuals that are surprising given the associated SHAP scores, and (3) P olyjuice -surprise , which uses the selection procedure described in the previous paragraph. We recruited 13 participants (graduate students with experience in model explanation), and had them analyze the aforementioned QQP model. In each round, users were shown an example, the model prediction, and a SHAP explanation, as in Figure 3A. Users were instructed to create up to 10 counterfactuals in order to better understand model behavior around the example, for which model predictions were given (users created 6 on average). Finally, users simulated what the model would do on six counterfactuals (Hase and Bansal, 2020), two from each condition (in random order). Counterfactuals where users make mistakes are prefer-8 Details in Appendix C.1. able, as displaying these would add information that users do not already have. As shown in Figure 4, humans simulated model behavior on P olyjuice -surprise counterfactuals only slightly better than random guessing (45%  6%), i.e., these examples display model behavior that is surprising to users even after seeing explanations and creating their own counterfactuals. Expert-surprise also had a high error rate, but at a much higher cost: generating these for just 20 original instances took 1.52 hours of expert labor. While high error rates could be achieved with unrelated or nonsensical examples, all counterfactuals under evaluation were close to the original examples, when measured by syntactic tree edit (  1 . 0) or Levenshtein distance (  0 . 2), P olyjuice surprise being the closest on both. An independent rater labeled 95% of P olyjuice -surprise counterfactuals as likely written by a native speaker, in contrast to 85% for Expert-surprise , indicating that experts sometimes resorted to ungrammatical or nonsensical sentences to find surprising behaviors. Qualitatively, the study participants tended to create counterfactuals by perturbing the token with the highest weights (84% of their  x perturbed tokens in the top 15% quantile of weights), not gaining a real understanding of how the other tokens impact predictions. Participants also made a significant number of mistakes even for tokens they had inspected, e.g., a participant perturbed the example in Figure 3A by replacing help (cid:41) play with, yielding a Non-Duplicate model prediction. When faced with help (cid:41) find in Figure 3D, they incorrectly assumed the behavior would be the same. These results indicate that P olyjuice counterfactuals complement feature attribution explanations by displaying information that users often miss, even after they have manually explored the model behavior beyond explanations. Moreover, P olyjuice counterfactuals for this application were more surprising and fluent than Expert-surprise , despite being computed automatically. , perturbed H with [BLANK]  x H :  Two women are looking out the window. H :  Ten women are looking out the window. H :  More than one personwindow. f (  x ) N eutral C ontradiction E ntailment [BLANK] looking out the window. Figure 5: (A) An NLI case with a Neutral prediction (underlined f ( x ) are correct). P olyjuice generates counterfactual hypotheses conditioned on the negation control code. (B) Generalizing perturbations into patterns (Wu et al., 2020). The change DET (cid:41) no flips 92 . 8% of predictions from N eutral (cid:41) C ontradiction. While our use of P olyjuice has so far relied on automatic selection of counterfactuals, we show in this section how an analyst can benefit from multiple counterfactuals per x , make use of controlled generation for more advanced analysis, and extract general patterns from individual observations. Our use case is counterfactual error analysis (Wu et al., 2019) of RoBERTa finetuned on NLI (used in 3.1), although the techniques are generally applicable. There is a known correlation between the label Contradiction and hypotheses with negation in NLI datasets (Gururangan et al., 2018), which may cause models to fail on non-contradiction negations. We explore this in Figure 5A by generating counterfactual hypotheses for a random Neutral instance, conditioning only on the original x and the negation control code. While the first two counterfactuals display this failure mode, there is a surprising inconsistency in model behavior between not and n't. We note that manual analysis may not explore these three negation forms, and thus not surface this puzzling behavior. To verify if the pattern is widespread, we generate counterfactuals with the negation control code for a random set of instances correctly predicted as Neutral ( n = 895). To generalize individual changes into patterns, we extract frequent counterfactual templates with Tempura (Wu et al., 2020) (details in Appendix D.2), shown in Figure 5B. The top templates (in bold) show that the model flips , perturbed H with [BLANK]  x H :  Two women are looking out the window. its prediction from Neutral to Contradiction with roughly the same frequency (  43%) whether the negation word is not or n't, but flips much more frequently with a di  erent negation pattern where a determiner is replaced with no (92 . 8%). While these behaviors may be correct in some instances, they often are not ( e.g., Figure 5A), and thus would warrant further exploration, and potential mitigation strategies ( e.g., counterfactual training, 3). Tangentially, the impact of DET (cid:41) no might lead the analyst to explore the impact of perturbing the subject of hypotheses, which we do in Figure 6 by placing a [BLANK] on the subject rather than using a control code. This leads to the discovery of unstable and erroneous behaviors regarding quantifiers , which we analyze in more detail in Appendix D.1. Discussion. P olyjuice is a powerful tool for interactive analysis. Generating multiple counterfactuals per instance leads to insights that might be missed by manual analysis, and the steering provided by control codes and [BLANK] s allow for analyses that would be non-trivial to do manually (Wu et al., 2019) or with masked language models ( e.g., Figure 5B places negations in various parts of sentences, and Figure 6 replaces spans with other spans of varying lengths). Besides error analysis, an analogous interactive use of P olyjuice may be suitable for test creation (Ribeiro et al., 2020) and forms of data augmentation that are more controlled than what we presented in 3. Some prior work in training and evaluation relies on humans to generate counterfactuals from scratch (Gardner et al., 2020; Teney et al., 2020; Kaushik et al., 2020). Our experiments in 3 indicate that asking humans to label P olyjuice counterfactuals yields similar or better results at a lower cost, which motivates an exploration of a mixture of manual and semi-automated generation. Similarly, prior work on analysis relies on experts to create individual counterfactuals or perturbation functions (Wu et al., 2019; Ribeiro et al., 2020). In 5, we show that P olyjuice enhances current practice by generating multiple counterfactuals that might have been overlooked, and by providing abstractions that allow for new kinds of analyses. Prior work on automatically generating counterfactuals typically has a narrower scope in terms of the relationships x (cid:41)  x . For example, adversarial generators aim to maintain semantics while changing model predictions (Ribeiro et al., 2018; Iyyer et al., 2018; Li et al., 2021), whereas concurrent work to our own (Madaan et al., 2021; Ross et al., 2020) automatically generates  x that change predictions for explanation or analysis, with no constraints on semantics. However, as shown in 35, a mix of label-preserving and label-flipping counterfactuals generated by P olyjuice is quite useful for training, evaluation, explanation, and analysis. Further, general-purpose counterfactuals may lead to serendipitous discoveries (5), especially as P olyjuice is not fine-tuned to the target domain (and thus less liable to merely replicate what is already there). Finally, by allowing control through control codes and [BLANK] s, P olyjuice supports human-generator collaboration, where a person specifies desired changes ( e.g., perturb the sentence subject ). Such collaboration is hard to imagine using automatic generators with no control, or with coarser control through predefined style attributes or labels (Madaan et al., 2020; Malmi et al., 2020). To our knowledge, prior work on controlled generation (Keskar et al., 2019; Dathathri et al., 2020) does not address counterfactual generation. We propose P olyjuice , a general-purpose generator that produces fluent and diverse counterfactuals, allowing for control over the kinds and locations of perturbations. With simple, task-specific selection heuristics, P olyjuice supports various downstream tasks on di  erent domains, including counterfactual data augmentation, contrast set generation, counterfactual explanation, and error analysis. While P olyjuice is broadly applicable, it is not bias-free: control codes are pre-defined and certainly not exhaustive, and the model is fine-tuned on a collection of paired datasets where certain perturbations are more or less likely ( e.g., we observe that words with negative sentiment tend to be slightly more likely than positive ones in some contexts). Collecting naturally occurring counterfactuals is an important area of future research, as is the development of generators that allow for control even without a-priori control codes. Besides improving the generators, further work is needed to improve the value of counterfactuals. For example, while P olyjuice shows consistent gains across tasks in data augmentation, the improvements on some datasets are not as significant. This aligns with observations in prior work that even manual counterfactuals can be marginally ben-eficial (Kaushik et al., 2020; Huang et al., 2020), possibly because the original data is already diverse enough, or the perturbed signal in counterfactuals is too subtle to a  ect the model ( e.g., when only a single word is changed in a long sentence.) We hope to perform more thorough experiments on tuning the amount and the distribution of counterfactual augmentation, as well as other ways of incorporating counterfactuals, such as having explicit terms in the loss function for contrasting counterfactuals with original data (Teney et al., 2020), or other forms of contrastive learning. Although our applications all involved people, the human-P olyjuice collaboration in labeling and explanations could benefit from richer interaction mechanisms. We believe P olyjuice motivates future research on more expressive forms of counterfactual training, where users generate counterfactuals together with P olyjuice , and label counterfactual patterns rather than individual instances. Similarly, interactive explanations and analysis are exciting directions, especially as we develop new ways of selecting, presenting, and aggregating counterfactuals for various analysis objectives. Having noted these opportunities, we believe P olyjuice is already a powerful tool for counterfactual reasoning, in particular for tasks where people are directly involved. P olyjuice is opensource, and available at https://github.com/tongshuangwu/polyjuice . The work was supported by ONR grant N00014-18-1-2193, NSF RAPID grant 2040196, NSF award IIS-1901386, the University of Washington WRF / Cable Professorship, and the Allen Institute for Artificial Intelligence (AI2). We thank Jim Chen, Dianqi Li, Scott Lundberg, Hao Peng, Sameer Singh, Jiao Sun, Victor Zhong, and Sitong Zhou for their helpful comments, as well as our user study participants for their valuable input. Our work includes labeling counterfactuals on crowdsourcing platforms, as well as conducting user studies with graduate students. As detailed in Appendix B.1 and C.2, we compensated the MTurk workers $2.5 for  15 minutes of labeling, and the graduate students $20 for the user study (one hour), above the U.S. federal minimum wage. The studies are with IRB approval. We only finetune GPT-2 rather than training it from scratch, such that our compute costs are relatively low (around 8 hours for finetuning, Appendix A). All of our finetuning experiments involved finetuning RoBERTa on smaller datasets. More critically, with most of our demonstrated applications using a human-generator hybrid mechanism, we stress that the interaction between the two deserves careful consideration. It has long been reported that algorithms interacting with humans can negatively impact the human. 9 In our case, the concern might be that users can develop an over-reliance on P olyjuice (Bansal et al., 2021) and hastily accept its generations. Not only can this decrease users' creativity (Green et al., 2014), but it may bias their analysis process: as discussed in 7, P olyjuice generation is not exhaustive, and may favor some perturbation patterns over others in unpredictable ways. In the short term, we plan to highlight these limitations as part of the model documentation, while future research should identify interaction mechanisms, so as to ensure that P olyjuice or other counterfactual generators support humans, rather than hindering their performance.
abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain objective method result method method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method other objective other objective other abstain other other other other method objective abstain result abstain abstain abstain other abstain objective objective method other other other method abstain abstain method method abstain abstain abstain abstain abstain