Natural language processing models often exploit spurious correlations between task-independent features and labels in datasets to perform well only within the distributions they are trained on, while not generalising to different task distributions. We propose to tackle this problem by generating a debiased version of a dataset, which can then be used to train a debiased, off-the-shelf model, by simply replacing its training data. Our approach consists of 1) a method for training data generators to generate high-quality, label-consistent data samples; and 2) a filtering mechanism for removing data points that contribute to spurious correlations, measured in terms of z-statistics . We generate debiased versions of the SNLI and MNLI datasets, 1 and we evaluate on a large suite of debiased, out-of-distribution, and adversarial test sets. Results show that models trained on our debiased datasets generalise better than those trained on the original datasets in all settings. On the majority of the datasets, our method outperforms or performs comparably to previous state-of-the-art debiasing strategies, and when combined with an orthogonal technique, product-of-experts, it improves further and outperforms previous best results of SNLI-hard and MNLI-hard. Natural Language Processing (NLP) datasets inevitably contain biases that are unrelated to the tasks they are supposed to represent. These biases are usually artifacts of the annotation processes, task framing, or design decisions (Schwartz et al., 2017; Geva et al., 2019; Liu et al., 2021). Such biases often manifest as spurious correlations between simple features of the data points and their  Work done while at the Allen Institute for AI. labels (Gardner et al., 2021). Trained models can exploit these spurious correlations to correctly predict the labels of the data points within the same distributions as those they are trained on, but fail to generalise to other distributions within the same tasks. Consequently, the models risk modelling the datasets, but not the tasks (Gururangan et al., 2018; Poliak et al., 2018; McCoy et al., 2019; Schuster et al., 2019). We address this issue by adjusting existing dataset distributions to mitigate the correlations between task-independent features and labels. First, we train data generators that generate high quality data samples in the distribution of existing datasets (Section 2). Then, we identify a set of simple features that are known to be task-independent, and use the theoretical framework (i.e., z-statistics) proposed by Gardner et al. (2021) to measure correlations between those features and the labels (Sec-tion 3.1). Finally, we adjust the distribution of the generated samples by post-hoc filtering (Sec-tion 3.2) to remove the data points that contribute to high z-statistics with task-independent features, or finetuning the data generator (Section 4.1) to make such data points less likely. Unlike prior model-2660 centric approaches to mitigate spurious correlations (Belinkov et al., 2019a,b; Clark et al., 2019; He et al., 2019; Karimi Mahabadi et al., 2020) that define new training objectives or model architectures, our approach has the advantage of keeping the objective and the model fixed, as we only alter the training data. To evaluate our approach, we use the task of Natural Language Inference (NLI), which offers a wide range of datasets (including challenge datasets) for various domains. We generate debiased SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) distributions and evaluate the generalisability of models trained on them to out-of-distribution hard evaluation sets (Guru-rangan et al., 2018; McCoy et al., 2019), and the adversarial attack suite for NLI proposed by Liu et al. (2020b). Furthermore, we compare our method to strong debiasing strategies from the literature (Be-linkov et al., 2019b; Stacey et al., 2020; Clark et al., 2019; Karimi Mahabadi et al., 2020; Utama et al., 2020; Sanh et al., 2021; Ghaddar et al., 2021). Our results show that models trained on our debiased datasets generalise better than those trained on the original datasets to evaluation sets targeting hypothesis-only biases (by up to 2.8 percentage points) and syntactic biases (by up to 13.3pp), and to a suite of adversarial tests sets (by up to 4.2pp on average). Since our contributions are orthogonal to model-centric approaches, we show that when combined with product-of-experts (Karimi Mahabadi et al., 2020), our method yields further improvements and outperforms previous state-of-the-art results of SNLI-hard and MNLI-hard. Finally, we train stronger and larger pretrained language models with our debiased datasets, and demonstrate that the performance gain by our method generalises to these larger models. First, we need to train a data generator G to generate data samples automatically. Our goal for the data generator is to model the true distribution as well as possible so that we can generate valid and high-quality data samples. choose GPT-2 because it is a powerful and widely-used autoregressive language model, and it can be easily adapted to generated the premise, label, and hypothesis of an instance sequentially. Given an NLI dataset D 0 , the training objective is to minimise the following negative log-likelihood loss of generating the premise-label-hypothesis sequence, in that order: LMLE =  |D 0 | (cid:88) i =1 log p ( P ( i ) , l ( i ) , H ( i ) ) =  |D 0 | (cid:88) i =1 log p ( P ( i ) ) p ( l ( i ) | P ( i ) ) p ( H ( i ) | l ( i ) , P ( i ) ) , (1) where P ( i ) , l ( i ) and H ( i ) are the premise, label and hypothesis respectively. We find that samples generated by a generator trained with only LMLE often contain ungrammatical text or incorrect label. In this section, we introduce two techniques to improve data quality. We observe poor label consistency in samples generated by a generator trained with vanilla LMLE objective  given a generated sample (  P ,  H,  l ) , the label  l often does not correctly describe the relationship between  P and  H . To alleviate this issue, we apply unlikelihood training (Welleck et al., 2020) to make generating such label inconsistent instances less likely. First we perturb the label to construct negative samples ( P, H, l (cid:48) ) where l (cid:48) (cid:54) = l for each sample in the dataset. Then we apply a token-level unlikelihood objective on the hypothesis tokens: L consistency =  |D 0 | (cid:88) i =1 | H | ( i ) (cid:88) t =1 log(1  p ( H ( i ) t | l (cid:48) ( i ) , P ( i ) , H ( i ) <t )) . This objective decreases the probability of generating H when given an incorrect label l (cid:48) , hence improves the label consistency at generation time. where  is a hyperparameter that balances the two objectives. We can randomly sample from the trained generator to obtain a large amount of the synthetic data DG  G . We add a consistency filtering step (Lewis et al., 2021; Bartolo et al., 2021) to further improve the quality of the generated dataset. We train an NLI model M with the original dataset D 0 to filter out samples in which M has low confidence:  DG = { ( P, H, l )  DG | p M ( l | P, H ) >  } , where  is a confidence threshold. We found that the filtered out data samples generally had ungrammatical text or incorrect labels. We now define a method to reject samples that contribute to the high spurious correlations between task-independent features of the samples and their labels. Our approach is based on the theoretical framework proposed by Gardner et al. (2021) to measure these correlations, known as z-statistics . Our filtering method, called z-filtering (Section 3.2), will serve as the basis to construct debiased datasets in Section 4. As a first step towards addressing spurious correlations, we need to be able to quantify them. We start by selecting a set of task-independent features  features that give away the labels and allow models to exploit them without actually solving the task. For NLI, we choose the following features: 1 ) unigrams and bigrams; 2 ) hypothesis length and hypothesis-premise length ratio; 3 ) lexical overlap between hypothesis and premise; 4 ) the predictions of a BERT-base (Devlin et al., 2019) hypothesis-only model. 3 These features capture various biases identified in prior work, including contradiction word biases, lexical overlap bias (McCoy et al., 2019), and hypothesis-only bias (Gururangan et al., 3 See Appendix B for detailed descriptions of the features. 2018; Poliak et al., 2018). Note that our method does not rely on the specific choice of features, and one can easily add alternative features that should not be correlated with the labels. Following Gardner et al. (2021), we assume there should be no correlation between each of these features and the class labels. More formally, for any feature x from our feature set X , p ( l | x ) should be uniform over the class labels l . We define  p ( l | x ) = 1 n (cid:80) nj =1 l j to be the empirical expectation of p ( l | x ) over n samples containing x . Then we compute the standardised version of z-statistics to quantify its deviation from the uniform distribution for each feature x and label l : z  ( x, l ) =  p ( l | x )  p 0 (cid:112) p 0 (1  p 0 ) /n, (2) where p 0 is the probability of uniform distribution ( p 0 = 1 / 3 in NLI tasks with three labels). These z-statistics scores can be used to identify the most biased features for each label l  we select k features with the highest z-statistic to define the biased features set BD ( l ) . Table 12 shows examples of these biased features on SNLI. To mitigate the biases in the dataset, we propose z-filtering , an algorithm that iteratively selects and filters instances from a dataset D (cid:48) to build a debiased dataset Z . At each step, we find the set of biased features BZ ( l ) on the partially constructed Z . We then select a new batch of samples from D (cid:48) and filter out the samples that contain these biased features. This process is applied iteratively until it has exhausted all samples from D (cid:48) . It removes the samples that contribute to the spurious correlations in D (cid:48) , thus it finds a debiased subset Z ( D (cid:48) )  D (cid:48) . We denote the removed samples as Z  ( D (cid:48) ) . The full z-filtering algorithm is illustrated in Algorithm 1. Optionally, one can initialise Z with a seed dataset D seed . In this case, the samples from D (cid:48) are only added to Z when they do not contain the biased features of D seed . Thus it can be seen as a data-augmentation technique targeted to debias a given dataset. We refer to it as conditional z-filtering and denote the produced debiased dataset as Z ( D (cid:48) |D seed ) . We use z-filtering in two ways: 1 ) to further finetune G (the one trained in Section 2.2.1 with consistency unlikelihood) with an objective that downweighs samples that should be rejected (Sec-tion 4.1); 2 ) to post-hoc filter the generated samples to obtain debiased datasets (Section 4.2). The generator G can learn to exploit task-independent features during its finetuning stage (Section 2), causing the synthetic data  DG to contain many spurious correlations. While it is tempting to apply z-filtering to remove these spurious correlations from  DG , we find that this will lead to the removal of majority of the generated data. For example, when the generator is finetuned on SNLI, z-filtering removes around 85% of  DGSNLI . 4 This leads to a very inefficient data generation process to mitigate the spurious correlations. To alleviate this issue, we can incorporate the debiasing objectives into the training of the generator, so that the samples produced by the generator 4 This is also strong confirmation that these biases are problematic, as the generative model easily finds them and relies on them during data generation. Conducting naive data augmentation with  DGSNLI will strengthen the spurious correlations. are more likely to be accepted by the z-filtering process. More specifically, we can encourage the model to generate Z ( D 0 ) , while discouraging it from generating Z  ( D 0 ) . For the latter part, we again apply an unlikelihood training objective LUL to unlearn Z  ( D 0 ) . Hence, the overall debiasing training objective is: L debias = LMLE ( Z ( D 0 )) +  LUL ( Z  ( D 0 )) where  is a hyperparameter. A naive use of an unlikelihood objective on all tokens gives the model mixed signals for good tokens and leads to ungrammatical, degenerate outputs. To avoid this degeneracy, we apply the unlikelihood loss only to tokens that contribute to biased features. Concretely, for each token I  t of instance I   Z  ( D 0 ) , we define a mask m t as m t = (cid:40) 0 , if I  t contributes to BZ ( l I  ) 1 , otherwise . where BZ ( l I  ) represent the biased features corresponding the label of I  . For biases towards unigram and bigram features (as defined in Section 3.1), we consider only the corresponding tokens to be relevant (i.e., m t = 0 if I  t is part of the unigram or the bigram). For biases towards other features (e.g. length of the hypothe-sis), we consider all the tokens on the hypothesis to be relevant. The unlikelihood training objective is defined as follows: LUL ( Z  ( D 0 )) = (cid:88) I (cid:48) Z  ( D 0 ) LUL ( I (cid:48) ) , LUL ( I (cid:48) ) =  | I (cid:48) | (cid:88) t =1 log( m t p ( I (cid:48) t | I (cid:48) <t ) +(1  m t )(1  p ( I (cid:48) t | I (cid:48) <t ))) . We further finetune G with L debias to obtain a new generator G  , that is trained to generate more unbiased data samples. We then randomly sample from G  and conduct data filtering (Section 2.2.2) to obtain a large set of high-quality debiased data samples  DG  . Given the original dataset D 0 and the synthetic dataset  DG  , our goal is produce a large-scale unbiased dataset D  . There are various ways to do 2663 this given that we can either apply conditional z-filtering, or simply z-filter both D 0 and  DG  and merge them. We explore the following options: 1. Z-Augmentation (Z-Aug) Z (  DG  |D 0 ) : we keep the original dataset as is, and augment it by conducting conditional z-filtering on  DG  using D 0 as seed dataset. 2. Parallel z-filter (Par-Z) Z ( D 0 )  Z (  DG  ) : we conduct z-filtering on D 0 and  DG  separately, and then merge them. 3. Sequential z-filter (Seq-Z) Z (  DG  |Z ( D 0 )) : we first conduct z-filtering on D 0 , then conduct conditional z-filtering on  DG  with Z ( D 0 ) as seed dataset. Source Datasets We select the two most widely used NLI datasets SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) as our original datasets. Prior work (Gururangan et al., 2018; Poliak et al., 2018; McCoy et al., 2019) found various annotation artifacts in them, hence they serve as good use cases for constructing debiased datasets. Evaluation Datasets For the hypothesis-only bias, we use the challenge sets SNLI-hard (Gu-rurangan et al., 2018) and MNLI-hard (Williams et al., 2018), which were produced by filtering the test set with a hypothesis-only model (Sec-tion 5.2). For syntactic biases, we follow previous work and use HANS (McCoy et al., 2019) for evaluation (Section 5.3). In addition, we evaluate on the adversarial test benchmark introduced by Liu et al. (2020b) (Section 5.4). This benchmark covers a wide range of adversarial attacks, which will give a more complete picture of what spurious correlations the debiasing methods tackle. Generating Debiased Datasets We conduct debiased data generation for SNLI and MNLI separately . For SNLI, we use the proposed method described in Section 4.1 to train a generator G  SNLI . Then we randomly sample a large number of instances from the generator to construct DG  SNLI . The samples are filtered with a strong NLI model M trained on SNLI to obtain  DG  SNLI . Finally, different options (Section 4.2) can be adopted to merge the synthetic data with the original data DSNLI to construct debiased versions of SNLI. The same Options D 0 = DSNLID 0 = DMNLI Original D 0 549,367 382,702 Z-Aug Z (  DG  |D 0 ) 1,142,475 744,326 Par-Z Z ( D 0 )  Z (  DG  ) 933,085 740,811 Seq-Z Z (  DG  |Z ( D 0 )) 927,906 744,200 Table 1: Data size of the constructed debiased datasets for SNLI and MNLI. procedure is used to produce debiased datasets for MNLI, by simply replacing the original dataset with MNLI. We choose GPT-2 large and Roberta-large as the pretrained language models for G  and M respectively. 5 The size of the constructed debiased datasets are listed in Table 1. NLI Model Training Since our method directly debiases the training data itself, we keep the model and training objective fixed and only replace the training data with our generated debiased datasets. For comparability with previous work (Karimi Mahabadi et al., 2020; Utama et al., 2020; Sanh et al., 2021), we train BERT-base (Devlin et al., 2019) on our debiased datasets. The NLI models are trained with ordinary cross-entropy classification loss, and the training hyperparameters are listed in Appendix A. We run our experiments five times and report the average and standard deviation of the scores. 6 We also conduct statistical significance testing using a 2-tailed t-test at 95% confidence level. State-of-the-art Debiasing Models We compare our method with the following three state-of-the-art debiasing models on each of our evaluation datasets. Product-of-Experts (He et al., 2019; Karimi Mahabadi et al., 2020) ensembles a bias-only model's prediction b i with the main model's p i using p (cid:48) i = softmax (log p i + log b i ) . This ensembling enforces that the main model focuses on the samples that the bias-only model does not predict well. Learned-Mixin (Clark et al., 2019) is a variant of PoE that introduces a learnable weight for the bias-only model's prediction. Regularized-conf (Utama et al., 2020) uses confidence regu-larisation to retain the in-distribution performance while conducting model debiasing. 5 On one A100 GPU, training the generator takes around 24 hours and generating the samples takes roughly 35 hours for each dataset. 6 With the exception of our PoE experiments which single run, as hyperparameter tuning for PoE is costlier. Combining PoE with Our Debiased Datasets Our approach changes the training data distribution instead of the model's training objective, and hence is orthogonal to prior work method-wise. We also report the results of combining PoE with our proposed method, simply by training a PoE model on our debiased datasets. We adapt the PoE implementation by Karimi Mahabadi et al. (2020), and we follow their approach to conduct hyperparameter tuning for PoE. 7 The hyperparameters of the PoE models are reported in Table 10 of Appendix A. 5.2 Hypothesis-only Bias in NLI Gururangan et al. (2018) found that, on SNLI and MNLI, a model that only has access to the hypothesis can perform surprisingly well, which indicates that the datasets contain hypothesis-only bias. To alleviate this problem, SNLI-hard and MNLI-hard (Gururangan et al., 2018) subsets were constructed by filtering the test set with a hypothesis-only model and only accepting those that the hypothesis-only model predicts incorrectly. We examine whether our method successfully mitigates the hypothesis-only bias in NLI, by evaluating the models trained with our debiased datasets on SNLI-hard and MNLI-hard. Results on SNLI-hard Table 2 shows the results of our method on SNLI and SNLI-hard. The results show that, compared to training on SNLI, training with our debiased datasets significantly improves the performance on SNLI-hard. The 7 https://github.com/rabeehk/robust-nli debiased dataset produced by Seq-Z achieves a 2.48% gain in accuracy on SNLI-hard compared to the SNLI baseline, whereas Z-Aug improves both SNLI and SNLI-hard accuracy. Results on MNLI-hard Table 3 shows the results of our method on MNLI-matched (MNLI-m) and MNLI-mismatched (MNLI-mm), and their corresponding hard sets. We use the development sets of MNLI-hard reconstructed by (Karimi Mahabadi et al., 2020) to develop our methods. To comply with the submission limit of MNLI leaderboard system, we select the best checkpoint among the five runs using the development set, and report its test set performance in Table 3. The results show that BERT-base models trained on our debiased MNLI datasets outperform the models trained on the original MNLI by a large margin on the MNLI-hard sets. In particular, the Z-Aug version of the debiased datasets gives a 2.72% and 2.76% gain in accuracy on MNLI-m hard and MNLI-mm hard respectively, and outperforms the previous state-of-the-art on MNLI-m, MNLI-mm, and MNLI-mm hard. Combining PoE with Our Debiased Datasets We investigate the combination of our method and PoE, to see if the two orthogonal techniques can work together to achieve better performance. Since hyperparameter tuning of PoE is costly, we choose the best version of the debiased dataset (Seq-Z for SNLI and Z-Aug for MNLI) using the development set accuracy, and train PoE with it. The results are listed in the last rows of Table 2 and Table 3. We can find that, on both SNLI and MNLI, combining PoE with our debiased dataset yields further improvements on SNLI-hard, MNLI-m hard, and MNLI-mm hard, outperforming previous state-of-the-art results on all three datasets. McCoy et al. (2019) show that NLI models trained on MNLI can exploit syntactic heuristics present in the data, such as lexical overlap, subsequence, and constituent features. They introduce HANS, an evaluation dataset that contains examples where the syntactic heuristics fail. To test whether our method mitigates the syntactic biases in NLI, we evaluate models trained on our debiased datasets on HANS. If our debiased dataset contains less syntactic bias than the original dataset, the model would not exploit the syntactic heuristics and thus perform better on HANS. Due to the high variance 2665 Method (model w/ data) MNLI-m MNLI-mm MNLI-m hard MNLI-mm hard dev test dev test dev test dev test Prior debiasing strategies trained on MNLI PoE (Karimi Mahabadi et al., 2020)  84.58 84.11 84.85 83.47 78.02 76.81 79.23 76.83 Learned-Mixin (Clark et al., 2019)  80.5 79.5 81.2 80.4 -79.2 -78.2 Regularized-conf (Utama et al., 2020)  84.6 84.1 85.0 84.2 -78.3 -77.3 BERT-base Main PoE+CE (Sanh et al., 2021)  83.32 -83.54 -77.63 -76.39 BERT-base w/ DMNLI baseline 83.87 84.11 84.22 83.51 76.39  0 . of the scores on HANS, we run five times for each experiment (except PoE), and report the average and standard deviation of the scores. Results on HANS Table 4 shows the results on HANS. The results are categorised into three sections according to the training data: SNLI, MNLI, and our debiased datasets. The results of models trained on our debiased MNLI datasets show strong improvements: compared to the original MNLI, our debiased MNLI datasets obtain up to a 13.33% gain in HANS accuracy. Our Seq-Z variant achieves 67.69% accuracy, which is comparable with strong PoE baseline (Karimi Mahabadi et al., 2020; Sanh et al., 2021). Our method also further improves PoE models: the BERT-base PoE model trained on our Z-Aug MNLI outperforms the one trained on MNLI by 5.3%. Additionally, training Roberta-large (Liu et al., 2019) on our debiased dataset introduces 2.9 points accuracy gain on HANS, indicating that the performance gain by our debiased dataset can generalise to larger and stronger models (more on this in Section 5.5). Liu et al. (2020b) find that debiasing methods often tie to one particular known bias and it is nontrivial to mitigate multiple NLI biases at the same time. They introduce a suite of test datasets for NLI models that targets various aspects of robustness, including partial input heuristics (PI), logical infer-2666 PI-CD PI-SP IS-SD IS-CS LI-LI LI-TS ST Avg. ence ability (LI), and stress test (ST). 8 Several data augmentation strategies were investigated by Liu et al. (2020b): 1) text swap: swapping the premise and hypothesis in the original data; 2) word substitution: replacing words in the hypothesis with synonyms or generations from a masked language model; 3) paraphrase: using back translation to paraphrase the hypothesis. We compare our approach with their data-augmentation heuristics, and the results are shown in Table 5. Comparing with the MNLI baseline, our debiased MNLI datasets lead to better performance across all categories, which indicates that our method successfully mitigates various distinct biases simultaneously. All three variants of our debiased datasets outperform the data augmentation heuristics by Liu et al. (2021), which demonstrates the efficacy of our method when compared against manually designed heuristics. Since our method mitigates the spurious correlations in the dataset, not the model, our approach is model-agnostic and has the potential to bene-fit larger future models. To test this hypothesis, we train stronger and more modern models than BERT with our debiased datasets, and see if it can still improve the performance. More specifically, we choose Roberta-base, Roberta-large (Liu et al., 2019), and Albert-xxlarge (Lan et al., 2020), train them with Seq-Z SNLI and Z-Aug MNLI. (2021); Bowman (2021); 2) training on our debiased datasets can still improve the performance of these models, yielding an average 2.30%, 1.23%, 1.13% gain for Roberta-base, Roberta-large and Albert-xxlarge respectively. This indicates that our method generalises to larger pretrained language models and could potentially enhance future models. Spurious Correlations in Datasets The issue of spurious correlations in datasets between labels and simple input features has recently received signifi-cant attention (Gururangan et al., 2018; Poliak et al., 2018; Belinkov et al., 2019a; Karimi Mahabadi et al., 2020). It has been shown that this issue is often inherent in the data annotation process, caused by biases in the framing of the task (Schwartz et al., 2017), noisy annotations (Chen et al., 2016), or personal (Geva et al., 2019) or group-level (Liu et al., 2021) annotator biases. Gardner et al. (2021) provide a theoretical framework for analyzing spurious correlations, which we use to define our filtering mechanism in Section 3.2. Debiasing NLI Models Much prior work follows a model-centric approach towards mitigating biases in NLI models  they propose novel model architectures or training objectives to ensure that the models do not exploit the shortcuts presented by the dataset biases. At the representation level, Belinkov et al. (2019a,b) introduce an adversarial architecture to debias hypothesis representations to tackle hypothesis-only bias (Gururangan et al., 2018), and Stacey et al. (2020) strengthen the debiasing by using multiple adversarial classifiers. Zhou and Bansal (2020) use HEX projection to project the representation to the space orthogonal to the biased features to debias the model. At the model level, Clark et al. (2019); He et al. (2019); Karimi Mahabadi et al. (2020) propose methods based on Product-of-Expert (PoE) (Hinton, 2002) for mitigating biases by ensembling a biased-only model with a main model. Utama et al. (2020) propose the use of confidence regularization to improve out-of-distribution performance while retaining in-distribution accuracy. Debiasing NLI Datasets Ross et al. (2021) introduce TAILOR, a semantically-controlled perturbation method for data augmentation based on a small number of manually defined perturbation strategies. Bras et al. (2020) propose AFLite, a dataset filtering method that learns feature representations with a model and conduct adversarial filtering based on model predictions. Unlike these approaches, our method requires no manually-written perturbation heuristics and is model-agnostic, hence it is more generally applicable. Generative Data Augmentation Several works investigate generative data augmentation techniques to improve model robustness in other areas. Yang et al. (2020) conduct generative data augmentation for commonsense reasoning and show that it can improve out-of-domain generalisation. Lee et al. (2021) trains a generator to generate new claims and evidence for debiasing fact verification datasets like FEVER (Thorne et al., 2018). Schick and Schutze (2021) exploit large pretrained language models to generate semantic textual similarity datasets. Bartolo et al. (2021) improve robustness of question answering models by generating adversarial dataset. To address the issue of spurious correlations between task-independent features and labels in NLI datasets, we propose methods to generate label-consistent data and then filter out instances from existing datasets that contribute to those spurious correlations; thereby generating debiased datasets. Models trained on our debiased versions of the SNLI and MNLI datasets generalise better than the equivalent model trained on the original datasets to a large suite of test sets focusing on various kinds of known biases. Future work in this direction includes investigating whether our techniques are applicable to tasks beyond NLI. The authors would like to thank Max Bartolo, Alexis Ross, Doug Downey, Jesse Dodge, Pasquale Minervini, and Sebastian Riedel for their helpful discussion and feedback.
abstain objective objective method result result abstain abstain abstain abstain abstain abstain method objective abstain objective abstain method abstain method result abstain objective abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain method method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method other abstain abstain abstain method abstain abstain method abstain method abstain method method method abstain method method method abstain abstain abstain abstain method abstain method method method method method abstain abstain abstain abstain abstain method method method method abstain abstain method method method other method method method method abstain method method abstain method abstain abstain abstain abstain other abstain abstain abstain result result result abstain abstain abstain abstain abstain result abstain abstain method abstain method abstain abstain other other method other other other other other other other abstain other other other other other objective method objective other