While pretrained language models achieve excellent performance on natural language understanding benchmarks, they tend to rely on spurious correlations and generalize poorly to out-of-distribution (OOD) data. Recent work has explored using counterfactually-augmented data (CAD)data generated by minimally perturbing examples to flip the ground-truth labelto identify robust features that are invariant under distribution shift. However, empirical results using CAD during training for OOD generalization have been mixed. To explain this discrepancy, through a toy theoretical example and empirical analysis on two crowdsourced CAD datasets, we show that: (a) while features perturbed in CAD are indeed robust features, it may prevent the model from learning unperturbed robust features; and (b) CAD may exacerbate existing spurious correlations in the data. Our results thus show that the lack of perturbation diversity limits CAD's effectiveness on OOD generalization, calling for innovative crowdsourcing procedures to elicit diverse perturbation of examples. Large-scale datasets have enabled tremendous progress in natural language understanding (NLU) (Rajpurkar et al., 2016; Wang et al., 2019) with the rise of pretrained language models (Devlin et al., 2019; Peters et al., 2018). Despite this progress, there have been numerous works showing that models rely on spurious correlations in the datasets, i.e. heuristics that are effective on a specific dataset but do not hold in general (McCoy et al., 2019; Naik et al., 2018; Wang and Culotta, 2020). For example, BERT (Devlin et al., 2019) trained on MNLI (Williams et al., 2018) learns the spurious correlation between world overlap and entailment label. A recent promising direction is to collect counterfactually-augmented data (CAD) by asking humans to minimally edit examples to flip their ground-truth label (Kaushik et al., 2020). Figure 1 shows example edits for Natural Language Inference (NLI). Given interventions on robust features that cause the label to change, the model is expected to learn to disentangle the spurious and robust features. Despite recent attempt to explain the efficacy of CAD by analyzing the underlying causal structure of the data (Kaushik et al., 2021), empirical results on out-of-distribution (OOD) generalization using CAD are mixed. Specifically, Huang et al. (2020) show that CAD does not improve OOD generalization for NLI; Khashabi et al. (2020) find that for question answering, CAD is helpful only when it is much cheaper to create than standard examples  but Bowman et al. (2020) report that the cost is actually similar per example. In this work, we take a step towards bridging this gap between what theory suggests and what we observe in practice in regards to CAD. An intuitive example to illustrate our key observation is shown in Figure 1 (a), where the verb eating' is changed to drinking' to flip the label. While there are many other words that could have been changed to flip the label, given only these two examples, the model learns to use only the verbs (e.g. using a Naive Bayes model, all other words would have zero weights). As a result, this model would fail when evaluated on examples such as those in (b) where the quantifier two' is changed to three', while a model trained on the unaugmented data may learn to use the quantifiers. First, we use a toy theoretical setting to formalize counterfactual augmentation, and demonstrate that with CAD, the model can learn to ignore the spurious features without explicitly intervening on them. However, we find that without perturbing all robust features to generate CAD, perturbations of one robust feature can prevent the model from learning other unperturbed robust features. Motivated by this, we set up an empirical analysis on 3668 Premise : The lady is standing next to her two children who are eating a pizza. two crowdsourced CAD datasets collected for NLI and Question Answering (QA). In the empirical analysis, we identify the robust features by categorizing the edits into different perturbation types (Wu et al., 2021) (e.g. negating a sentence or changing the quantifiers), and show that models do not generalize well to unseen perturbation types, sometimes even performing worse than models trained on unaugmented data. Our analysis of the relation between perturbation types and generalization can help explain other observations such as CAD being more beneficial in the low-data regime. With increasing data size, improvement from using CAD plateaus compared to unaugmented data, suggesting that the number of perturbation types in existing CAD datasets does not keep increasing. Another consequence of the lack of diversity in edits is annotation artifacts, which may produce spurious correlations similar to what happens in standard crowdsourcing procedures. While CAD is intended to debias the dataset, surprisingly, we find that crowdsourced CAD for NLI exacerbates word overlap bias (McCoy et al., 2019) and negation bias (Gururangan et al., 2018a) observed in existing benchmarks. In sum, we show that while CAD can help the model ignore spurious feature, its effectiveness in current CAD datasets is limited by the set of robust features that are perturbed. Furthermore, CAD may exacerbate spurious correlations in existing benchmarks. Our results highlight the importance of increasing the diversity of counterfactual perturbations during crowdsourcing: We need to elicit more diverse edits of examples that make models more robust to the complexity of language. In this section, we use a toy setting with a linear Gaussian model and squared loss to formalize counterfactual augmentation and discuss the conditions required for it's effectiveness. The toy example serves to motivate our empirical analysis in Section 3. We adopt the setting in Rosenfeld et al. (2021): each example consists of robust features x r  R d r whose joint distribution with the label is invariant during training and testing, and spurious features x s  R d s whose joint distribution with the label varies at test time. Here d r and d s denote the feature dimensions. We consider a binary classification setting where the label y  { 1 , 1 } is drawn from a uniform distribution, and both the robust and spurious features are drawn from Gaussian distributions. Specifically, an example x = [ x r , x s ]  R d is generated by the following process (where d = d r + d s ): y = (cid:40) 1 w.p. 0 . where  r  R d r ;  s  R d s ;  r ,  s  R ; and I is the identity matrix. 1 The corresponding data distribution is denoted by D . Note that the relation between y and the spurious features x s depends on  s and  s , which may change at test time, thus relying on x s may lead to poor OOD performance. 1 This model corresponds to the anti-causal setting (Scholkopf et al., 2012), i.e. the label causing the features. We adopt this setting since it is consistent with how most data is generated in tasks like NLI, sentiment analysis etc. 3669 Intuitively, in this toy setting, a model trained with only access to examples from D would not be able to differentiate between the spurious and robust features, since they play a similar role in the data generating process for D . Formally, consider the setting with infinite samples from D where we learn a linear model ( y = w T x where w  R d ) by least square regression. Let  w  R d be the optimal solution on D (without any counterfactual augmentation). The closed form solution is: Cov( x, x )  w = Cov( x, y )  w = Cov( x, x )  1  (4) where  = [  r ,  s ]  R d and Cov(  ) denotes the covariance matrix: Cov( x, x ) = (cid:20)  r  r  Ts  s  Tr  s (cid:21) , (5) where  r ,  s are covariance matrices of x r and x s respectively. This model relies on x s whose relationship with the label y can vary at test time, thus it may have poor performance under distribution shift. A robust model w inv that is invariant to spurious correlations would ignore x s : w inv = (cid:2)   1 r  r , 0 (cid:3) . The counterfactual data is generated by editing an example to flip its label. We model the perturbation by an edit vector z that translates x to change its label from y to  y (i.e. from 1 to -1 or vice versa). For instance, the counterfactual example of a positive example ( x, +1) is ( x + z,  1) . Specifically, we define the edit vector to be z = [ yz r , yz s ]  R d , where z r  R d r and z s  R d s are the displacements for the robust and spurious features. Here, z is label-dependent so that examples with different labels are translated in opposite directions. Therefore, the counterfactual example ( x c ,  y ) generated from ( x, y ) has the following distribution: x cr |  y  N ( y (  r + z r ) ,  2 r I ) , (7) x c s |  y  N ( y (  s + z s ) ,  2 s I ) . Optimal edits. Ideally, the counterfactual data should de-correlate x s and y , thus it should only perturb the robust features x r , i.e. z = [ yz r , 0] . To find the displacement z r that moves x across the decision boundary, we maximize the log-likelihood of the flipped label under the data generating distribution D : z  r = arg max z r  R dr E ( x,y ) D log p (  y | x + [ yz r , 0]) =  2  r . Intuitively, it moves the examples towards the mean of the opposite class along coordinates of the robust features. Using the edits specified above, if the model trained on D c has optimal solution  w c , we have: Cov( x, x )  w c = Cov( x, y )  w c = (cid:2)   1 r  r , 0 (cid:3) = w inv . Incomplete edits. There is an important assump-tion made in the above result: we have assumed all robust features are edited. Suppose we have two sets of robust features x r 1 and x r 2 , 2 then not editing x r 2 would effectively make it appear spurious to the model and indistinguishable from x s . In practice, this happens when there are multiple robust features but only a few are perturbed during counterfactual augmentation (which can be common during data collection if workers rely on simple patterns to make the minimal edits). Considering the NLI example, if all entailment examples are flipped to non-entailment ones by inserting a negation word, then the model will only rely on negation to make predictions. More formally, consider the case where the original examples x = [ x r 1 , x r 2 , x s ] and counterfactual examples are generated by incomplete edits z = [ z r 1 , 0 , 0] that perturb only x r 1 . Using the same analysis above where z r 1 is chosen by maximum likelihood estimation, let the model learned on the incompletely augmented data be denoted by  w inc . We can then show that the error of the model trained from incomplete edits can be more than that of the model trained without any counterfactual augmentation under certain conditions. More formally, we have the following: 2 We assume they are conditionally independent given the label. Proposition 1. Define the error for a model as (cid:96) ( w ) = E x F (cid:2) ( w T inv x  w T x ) 2 (cid:3) where the distribution F is the test distribution in which x r and x s are independent: x r | y  N ( y r ,  2 r I ) and x s  N (0 , I ) . Assuming all variables have unit variance (i.e.  r = 1 and  s = 1), (cid:107)  r (cid:107) = 1, and (cid:107)  s (cid:107) = 1 , we get (cid:96) (  w inc ) > (cid:96) (  w ) if (cid:107)  r 1 (cid:107) 2 < 1+  13 6  0 . 767 , where (cid:107)  (cid:107) denotes the Euclidean norm, and  r 1 is the mean of the perturbed robust feature r 1 . Intuitively, this statement says that if the norm of the edited robust features (in the incomplete-edits model) is sufficiently small, then the test error for a model with counterfactual augmentation will be more than a model trained with no augmentation. Proof Sketch. The proof mainly follows from algebra and using the fact that Cov( x, x )  1 is a block matrix consisting of rank-one perturbations of the identity matrix. We refer the reader to Appendix A for the detailed proof. Thus, Proposition 1 implies that perturbing only a small subset of robust features could perform worse than no augmentation, indicating the importance of diversity in CAD. Next, we show that the problem of incomplete edits is exhibited in real CAD too. In this section, we test the following hypothesis based on the above analysis: models trained on CAD are limited to the specific robust features that are perturbed and may not learn other unperturbed robust features. We empirically analyze how augmenting counterfactual examples by perturbing one robust feature affects the performance on examples generated by perturbing other robust features. Perturbation types. Unlike the toy example, in NLU it is not easy to define robust features since they typically correspond to the semantics of the text (e.g. sentiment). Following Kaushik et al. (2021) and similar to our toy model, we define robust features as spans of text whose distribution with the label remains invariant, whereas spans of text whose dependence on the label can change during evaluation are defined as spurious features. We then use linguistically-inspired rules (Wu et al., 2021) to categorize the robust features into several perturbation types : negation , quantifier , lexical , insert , delete and resemantic . Table 1 gives the definitions of each type. Train/test sets. Both the training sets and the test sets contain counterfactual examples generated by a particular perturbation type. To test the generalization from one perturbation type to another, we use two types of test sets: aligned test sets where examples are generated by the same perturbation type as the training data; and unaligned test sets where examples are generated by unseen perturbation types (e.g. training on examples from lexical and testing on negation ). Data. We experiment on two CAD datasets collected for SNLI (Kaushik et al., 2020) and BoolQ (Khashabi et al., 2020). The size of the paired data (seed examples and edited examples) for each perturbation type in the training dataset is given in Table 1. Since some types (e.g. delete ) contain too few examples for training, we train on the top three largest perturbation types: lexical , insert , and resemantic for SNLI; and lexical , negation , and resemantic for BoolQ. 2: Accuracy of NLI CAD on both aligned and unaligned test sets. We report the mean and standard deviation across 5 random seeds. Each dataset has a total of 1400 examples. On average models perform worse on unaligned test sets (i.e. unseen perturbation types). experiments, we use 700 seed examples and their corresponding 700 perturbations for each perturbation type. As a baseline (SNLI seed'), we subsample examples from SNLI to create a similar sized dataset for comparison. 3 For BoolQ (Clark et al., 2019a), our initial experiments show that training on only CAD does not reach above random-guessing. Thus, we include all original training examples in BoolQ (Khashabi et al., 2020), and replace part of them with CAD for each perturbation type. This results in a training set of 9427 examples of which 683 are CAD for each perturbation type. The size 683 is chosen to match the the smallest CAD type for BoolQ. As a baseline (BoolQ seed'), we train on all the original training examples, consisting of 9427 examples. For both datasets, the training, dev and test sets are created from their respective splits in the CAD datasets. The size of the dev and test sets is reported in Appendix B.2. Model. We use the Hugging Face implementation (Wolf et al., 2020) of RoBERTa (Liu et al., 2019) to fine-tune all our models. To account for the small dataset sizes, we run all our experiments with 5 different random seeds and report the mean and standard deviation. Details on hyperparameter tuning are reported in Appendix B.1. 4 3 We observe similar trends when using different subsets of the SNLI data. We discuss results for the main question in this sectionhow does adding CAD generated from one perturbation type affect performance on examples generated from other perturbation types? Table 2 and 3 show results for SNLI and BoolQ. CAD performs well on aligned test sets. We see that on average models perform very well on the aligned test sets (same perturbation type as the training set), but do not always do well on unaligned test sets (unseen perturbation types), which is consistent with our analysis in Section 2. On SNLI, one exception is resemantic , which performs well on unseen perturbation types. We believe this is because it is a broad category (replac-ing any constituent) that covers other types such as lexical (replacing any word). Similarly, on BoolQ, lexical and resemantic both perform better than the baseline on some unaligned test sets (e.g. quantifier ), but they perform much better on the aligned test sets. CAD sometimes performs worse than the baseline on unaligned test sets. For example, on SNLI, training on insert does much worse than the seed baseline on lexical and resemantic , and SNLI seed performs best on quantifier and negation . On BoolQ, training on negation does slightly worse than the baseline on lexical and resemantic . This suggests that augmenting perturbations of one particular robust feature may reduce the model's reliance on other robust features, 3672 ins ins+lex ins+lex+resem All Types Diversity 56 58 60 62 64 66 68 70 A cc u r a c y o n MNLI ( OOD ) SNLI Baseline Figure 2: OOD accuracy (mean, std. deviation) on MNLI of models trained on SNLI CAD and SNLI seed (baseline) with increasing number of perturbation types and fixed training set size. In Section 3.3, we have seen that training on CAD generated by a single perturbation type does not generalize well to unseen perturbation types. However, in practice CAD contains many different perturbation types. Do they cover enough robust features to enable OOD generalization? Increasing Diversity. We first verify that increasing the number of perturbed robust features leads to better OOD generalization. Specifically, we train models on subsets of SNLI CAD with increasing coverage of perturbation types and evaluate on MNLI as the OOD data. Starting with only insert , we add one perturbation type at a time until all types are included; the total number of examples are fixed throughout the process at 1400 (which includes 700 seed examples and the corresponding 700 perturbations). Figure 2 shows the OOD accuracy on MNLI when trained on CAD and SNLI seed examples of the same size. We observe that as the number of perturbation types increases, models generalize better to OOD data despite fixed training data size. The result highlights the importance of collecting a diverse set of counterfactual examples, even if each perturbation type is present in a small amount. A natural question to ask here is: If we continue to collect more counterfactual data, does it cover more perturbation types and hence lead to better OOD generalization? Thus we investigate the im-BERT RoBERTa SNLI seed 59.7 0.3 73.8 1.2 CAD 60.2 1.0 70.0 1.1 Table 4: Accuracy (mean and std. deviation across 5 runs) on MNLI of different pretrained models fine-tuned on SNLI seed and CAD. Role of Dataset Size. To better understand the role dataset size plays in OOD generalization, we plot the learning curve on SNLI CAD in Figure 3, where we gradually increase the amount of CAD for training. The baseline model is trained on SNLI seed examples of the same size, and all models are evaluated on MNLI (as the OOD dataset). We also conduct a similar experiment on BoolQ in Figure 4, where a subset of MultiRC (Khashabi et al., 2018) is used as the OOD dataset following Khashabi et al. (2020). Since the test set is unbalanced, we report F1 scores instead of accuracy in this case. For SNLI, CAD is beneficial for OOD generalization only in low data settings ( < 2000 examples). As the amount of data increases, the comparable SNLI baseline performs better and surpasses the performance of CAD. Similarly for BoolQ, we observe that CAD is comparable to the baseline in the low data setting (  1000 examples). Surprisingly, more CAD for BoolQ leads to worse OOD performance. We suspect this is due to overfitting to the specific perturbation types present in BoolQ CAD. Intuitively, as we increase the amount of data, the diversity of robust features covered by the seed examples also increases. On the other hand, the benefit of CAD is restricted to the perturbed robust features. The plateaued performance of CAD (in the case of NLI) shows that the diversity of perturbations may not increase with the data size as fast as we would like, calling for better crowdsourcing protocols to elicit diverse edits from workers. Role of Pretraining. Tu et al. (2020) show that larger pretrained models generalize better from minority examples. Therefore, in our case we would expect CAD to have limited benefit on larger pretrained models since they can already leverage the 5 The results in Figure 2 when all perturbation types are included indicate that CAD performs better than the SNLI baseline. This is not in contradiction with the results found in Huang et al. (2020), since our models are trained on only a subset of CAD. This further motivates the study of how CAD data size affects generalization. diverse (but scarce) robust features revealed by SNLI examples. We compare the results of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) trained on SNLI CAD in Table 4  both models are fine-tuned on the SNLI CAD dataset and are evaluated on the OOD set (MNLI). For the RoBERTa model (pretrained on more data), CAD no longer improves over the SNLI baseline, suggesting that current CAD datasets may not have much better coverage of robust features than what stronger pretrained models can already learn from benchmarks like SNLI. An artifact of underdiverse perturbations is the newly introduced spurious correlations. As an example, in the extreme case where all entailment examples are flipped to non-entailment by the negation operation in Table 1, the model would learn to exclusively rely on the existence of negation words to make predictions, which is clearly undesirable. In this section, we study the impact of CAD on two known spurious correlations in NLI benchmarks: word overlap bias (McCoy et al., 2019) and negation bias (Gururangan et al., 2018b). Negation bias. We take examples where there is a presence of a negation word (i.e. "no", "not", "n't") in the hypothesis, and plot the fraction of examples in each class in both the seed and the Stress Test MNLI subset SNLI Seed 57.5 4.6 63.3 3.8 CAD 49.6 1.5 55.7 4.2 Table 5: Accuracy of models on challenge examples in the stress test and MNLI, where non-contradiction examples contain a negation word in the hypothesis. corresponding CAD examples in Figure 5a. As expected, contradiction is the majority class in the seed group, but surprisingly, including CAD amplifies the fraction of contradiction examples! As a result, training on CAD leads to worse performance on challenge sets that counter the negation bias compared to training on seed examples of the same size. Specifically, we test on the negation' part of the Stress Tests (Naik et al., 2018) 6 and challenge examples in the combined MNLI development set which contain negation words in the hypothesis but are not contradictions. Table 5 shows that models trained on CAD perform worse on both test sets, implying that they rely more on the negation bias. Word-overlap bias. Similarly, in Figure 5b, we show that CAD amplifies the fraction of entailment examples among those with high word overlap (i.e. more than 90% of words in the hypoth-6 Synthetic examples where the phrase and false is not true is appended to the hypothesis of MNLI examples. esis are present in the premise). Models trained on SNLI and CAD both perform poorly (< 10% accuracy) on the non-entailment subset of HANS challenge set (McCoy et al., 2019), which exploits the word overlap bias. Takeaway. This section reveals that in the process of creating CAD, we may inadvertently exacerbate existing spurious correlations. The fundamental challenge here is that perturbations of the robust features are only observed through word change in the sentenceit is hard to surface the underlying causal variables without introducing (additional) artifacts to the sentence form. Label-Preserving Data Augmentation. A common strategy to build more robust models is to augment existing datasets with examples similar to those from the target distribution. Min et al. (2020) improve accuracy on HANS challenge set (McCoy et al., 2019) by augmenting syntactically-rich examples. Jia and Liang (2016) and Andreas (2020) recombine examples to achieve better compositional generalization. There has also been a recent body of work using task-agnostic data augmentation by paraphrasing (Wei and Zou, 2019), back-translation (Sennrich et al., 2016) and masked language models (Ng et al., 2020). The main difference between these works and CAD is that the edits in these works are label-preserving whereas they are label-flipping in CADthe former prevents models from being over-sensitive and the latter alleviates under-sensitivity to perturbations. Label-Changing Data Augmentation. Lu et al. (2020) and Zmigrod et al. (2019) use rule-based CAD to mitigate gender stereotypes. Gardner et al. (2020) build similar contrast sets using expert edits for evaluation. In contrast, Kaushik et al. (2020) crowdsource minimal edits. Recently, Teney et al. (2020) also use CAD along with additional auxiliary training objectives and demonstrate improved OOD generalization. Kaushik et al. (2021) analyze a similar toy model (linear Gaussian model) demonstrating the benefits of CAD, and showed that noising the edited spans hurts performance more than other spans. Our analysis complements theirs by showing that while spans identified by CAD are useful, a lack of diversity in these spans limit the effectiveness of CAD, thus better coverage of robust features could potentially lead to better OOD generalization. Robust Learning Algorithms. Another direction of work has explored learning more robust models without using additional augmented data. These methods essentially rely on learning debi-ased representationsWang et al. (2018) create a biased classifier and project its representation out of the model's representation. Along similar lines, Belinkov et al. (2019) remove hypothesis-only bias in NLI models by adversarial training. He et al. (2019) and Clark et al. (2019b) correct the conditional distribution given a biased model. Utama et al. (2020) build on this to remove unknown' biases, assuming that a weak model learns a biased representations. More recently, Veitch et al. (2021) use ideas from causality to learn invariant predic-3675 tors from counterfactual examples. The main difference between these methods and CAD is that the former generally requires some prior knowledge of what spurious correlations models learn (e.g. by constructing a biased model or weak model), whereas CAD is a more general human-in-the-loop method that leverages humans' knowledge of robust features. In this work, we first analyzed CAD theoretically using a linear model and showed that models do not generalize to unperturbed robust features. We then empirically demonstrated this issue in two CAD datasets, where models do not generalize well to unseen perturbation types. We also showed that CAD amplifies existing spurious correlations, pointing out another concern. Given these results, a natural question is: How can we fix these problems and make CAD more useful for OOD generalization? We discuss a few directions which we think could be helpful:  We can use generative models (Raffel et al., 2020; Lewis et al., 2020) to generate diverse minimal perturbations and then crowdsource labels for them (Wu et al., 2021). We can improve the diversity of the generations by masking different spans in the text to be in-filled, thus covering more robust features. An alternative to improving the crowdsourcing procedure is to devise better learning algorithms which mitigate the issues pointed out in this work. For example, given that we know the models do not always generalize well to unperturbed features, we can regularize the model to limit the reliance on the perturbed features. We hope that this analysis spurs future work on CAD, making them more useful for OOD generalization. We thank Divyansh Kaushik, Tatsunori Hashimoto and members of the NYU ML2 group for discussion and feedback on the work. The first author is supported by a NSF Graduate Research Fellowship under grant number 1839302. This work was partly supported by Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI).
abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain objective result objective abstain abstain abstain abstain abstain result result abstain result objective objective method abstain method abstain abstain abstain abstain abstain method method abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method other method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain method method method method abstain abstain method abstain abstain abstain method method abstain method method abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other abstain other other other other other other other other objective objective result result method result result method abstain other other other