summaries: This paper proposes to learn a subset of a given dataset that acts as an adversary, that hurts the model performance when used as a training dataset. The central claim of the paper is that existing datasets on which models are trained are potentially biased, and are not reflective of real world scenarios. By discarding samples that add to this bias, the idea is to make the model perform better in the wild. The authors propose a method to do so, and then refine it so that the resulting solution is tractable. They implement the method on several datasets and show that by finding these adversarial samples, they indeed hurt model performance.  This paper hypothesizes that even though we are able to achieve very impressive performance on benchmark datasets as of now (e.g. image net), it might be due to the fact that benchmarks themselves have biases. They introduce an algorithm that selects more representative data points from the dataset that allow to get a better estimate of the performance in the wild. The algorithm ends up selecting more difficult/confusing instances. the paper proposes an algorithm that adversarially filters out examples to reduce dataset-specific spurious bias. the key intuition is that the datasets are curated in a way that easy to obtain samples have higher probability to be admitted to the dataset. however, not all real world samples are easy to obtain. in other words, real world samples may follow a completely different distribution than curated samples with easy-to-obtain ones.
abstractText: Large-scale benchmark datasets have been among the major driving forces in AI, supporting training of models and measuring their progress. The key assumption is that these benchmarks are realistic approximations of the target tasks in the real world. However, while machine performance on these benchmarks advances rapidly — often surpassing human performance — it still struggles on the target tasks in the wild. This raises an important question: whether the surreal high performance on existing benchmarks are inflated due to spurious biases in them, and if so, how we can effectively revise these benchmarks to better simulate more realistic problem distributions in the real world. In this paper, we posit that while the real world problems consist of a great deal of long-tail problems, existing benchmarks are overly populated with a great deal of similar (thus non-tail) problems, which in turn, leads to a major overestimation of true AI performance. To address this challenge, we present a novel framework of Adversarial Filters to investigate model-based reduction of dataset biases. We discuss that the optimum bias reduction via AFOPTIMUM is intractable, thus propose AFLITE, an iterative greedy algorithm that adversarially filters out data points to identify a reduced dataset with more realistic problem distributions and considerably less spurious biases. AFLITE is lightweight and can in principle be applied to any task and dataset. We apply it to popular benchmarks that are practically solved — ImageNet and Natural Language Inference (SNLI, MNLI, QNLI) — and present filtered counterparts as new challenge datasets where the model performance drops considerably (e.g., from 84% to 24% for ImageNet and from 92% to 62% for SNLI), while human performance remains high. An extensive suite of analysis demonstrates that AFLITE effectively reduces measurable dataset biases in both the synthetic and real datasets. Finally, we introduce new measures of dataset biases based on K-nearest-neighbors to help guide future research on dataset developments and bias reduction.
authors.name: DATASET BIASES
references.authors: Michael A Alcorn Qi Li Zhitao Gong Chengfei Wang Long Mai Wei-Shinn Ku Anh Nguyen
references.title: Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Matej Balog Nilesh Tripuraneni Zoubin Ghahramani Adrian Weller
references.title: Lost relatives of the gumbel trick
references.venue: In ICML,
references.authors: Samuel R. Bowman Gabor Angeli Christopher Potts Christopher D. Manning
references.title: A large annotated corpus for learning natural language inference
references.venue: In EMNLP,
references.authors: Qian Chen Xiao-Dan Zhu Zhen-Hua Ling Si Wei Hui Jiang Diana Inkpen
references.title: Enhanced lstm for natural language inference
references.authors: Ekin D Cubuk Barret Zoph Dandelion Mane Vijay Vasudevan Quoc V Le
references.title: Autoaugment: Learning augmentation strategies from data
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.venue: In NAACL-HLT,
references.authors: Kevin Eykholt Ivan Evtimov Earlence Fernandes Bo Li Amir Rahmati Chaowei Xiao Atul Prakash Tadayoshi Kohno Dawn Xiaodong Song
references.title: Robust physical-world attacks on deep learning models
references.authors: David F Fouhey Wei-cheng Kuo Alexei A Efros Jitendra Malik
references.title: From lifestyle vlogs to everyday interactions
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Robert Geirhos Patricia Rubisch Claudio Michaelis Matthias Bethge Felix A Wichmann Wieland Brendel
references.title: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness
references.authors: Mor Geva Yoav Goldberg Jonathan Berant
references.title: Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding
references.authors: Max Glockner Vered Shwartz Yoav Goldberg
references.title: Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, July 2018
references.venue: Association for Computational Linguistics. doi: 10.18653/v1/P18-2103. URL https: //www.aclweb.org/anthology/P18-2103
references.authors: Yash Goyal Tejas Khot Douglas Summers-Stay Dhruv Batra Devi Parikh
references.title: Making the v in vqa matter: Elevating the role of image understanding in visual question answering
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Emil Julius Gumbel Julius Lieblein
references.title: Statistical theory of extreme values and some practical applications: A series of lectures, volume 33
references.venue: US Government Printing Office Washington,
references.authors: Suchin Gururangan Swabha Swayamdipta Omer Levy Roy Schwartz Samuel Bowman Noah A. Smith
references.title: Annotation artifacts in natural language inference data
references.venue: In NAACL-HLT,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Eric Jang Shixiang Gu Ben Poole
references.title: Categorical reparameterization with gumbel-softmax
references.venue: In ICLR,
references.authors: Robin Jia Percy Liang
references.title: Adversarial examples for evaluating reading comprehension systems
references.venue: In EMNLP,
references.authors: Carolyn Kim Ashish Sabharwal Stefano Ermon
references.title: Exact sampling with integer linear programs and random perturbations
references.venue: In AAAI,
references.authors: Wouter Kool Herke van Hoof Max Welling
references.title: Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement
references.authors: Yi Ci Li Nuno Vasconcelos
references.title: REPAIR: Removing representation bias by dataset resampling
references.authors: Yingwei Li Yi Li Nuno Vasconcelos
references.title: RESOUND: Towards action recognition without representation bias
references.venue: In ECCV,
references.authors: Nelson F. Liu Roy Schwartz Noah A. Smith
references.title: Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
references.authors: Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar S. Joshi Danqi Chen Omer Levy Mike Lewis Luke S. Zettlemoyer Veselin Stoyanov
references.title: RoBERTa: A robustly optimized BERT pretraining approach
references.venue: ArXiv, abs/1907.11692,
references.authors: Chris J. Maddison Andriy Mnih Yee Whye Teh
references.title: The concrete distribution: A continuous relaxation of discrete random variables
references.venue: In ICLR,
references.authors: Tom McCoy Ellie Pavlick Tal Linzen
references.title: Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics,
references.authors: Aakanksha Naik Abhilasha Ravichander Norman Sadeh Carolyn Rose Graham Neubig
references.title: Stress test evaluation for natural language inference
references.venue: In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics,
references.authors: Yixin Nie Adina Williams Emily Dinan Mohit Bansal Jason Weston Douwe Kiela
references.title: Adversarial NLI: A new benchmark for natural language understanding
references.authors: Jeffrey Pennington Richard Socher Christopher D. Manning
references.title: Glove: Global vectors for word representation
references.venue: In EMNLP,
references.authors: Matthew E. Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke S. Zettlemoyer
references.title: Deep contextualized word representations
references.authors: Adam Poliak Jason Naradowsky Aparajita Haldar Rachel Rudinger Benjamin Van Durme
references.title: Hypothesis only baselines in natural language inference
references.venue: In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics,
references.authors: Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang
references.title: Squad: 100, 000+ questions for machine comprehension of text
references.venue: In EMNLP,
references.authors: Keisuke Sakaguchi Ronan Le Bras Chandra Bhagavatula Yejin Choi
references.title: Winogrande: An adversarial winograd schema challenge at scale
references.authors: Mingxing Tan Quoc Le
references.title: Efficientnet: Rethinking model scaling for convolutional neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Antonio Torralba Alexei A. Efros
references.title: Unbiased look at dataset bias
references.venue: CVPR, pp. 1521–1528,
references.authors: Masatoshi Tsuchiya
references.title: Performance impact caused by hidden bias of training data for recognizing textual entailment
references.venue: In Proceedings of the 11th Language Resources and Evaluation Conference,
references.authors: Tim Vieira
references.title: Gumbel-max trick and weighted reservoir sampling, 2014
references.venue: Blog post
references.authors: Alex Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman
references.title: Glue: A multi-task benchmark and analysis platform for natural language understanding
references.venue: In ICLR,
references.authors: Adina Williams Nikita Nangia Samuel Bowman
references.title: A broad-coverage challenge corpus for sentence understanding through inference
references.venue: In NAACL-HLT,
references.authors: Rowan Zellers Yonatan Bisk Roy Schwartz Yejin Choi
references.title: SWAG: A large-scale adversarial dataset for grounded commonsense inference
references.venue: In EMNLP,
references.authors: Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi
references.title: HellaSwag: Can a machine really finish your sentence
sections.text: Large-scale benchmark datasets have been among the major driving forces in AI, supporting training of models and measuring their progress. The key assumption is that these benchmarks are realistic approximations of the target tasks in the real world. However, while machine performance on these benchmarks advances rapidly — often surpassing human performance — it still struggles on the target tasks in the wild. This raises an important question: whether the surreal high performance on existing benchmarks are inflated due to spurious biases in them, and if so, how we can effectively revise these benchmarks to better simulate more realistic problem distributions in the real world. In this paper, we posit that while the real world problems consist of a great deal of long-tail problems, existing benchmarks are overly populated with a great deal of similar (thus non-tail) problems, which in turn, leads to a major overestimation of true AI performance. To address this challenge, we present a novel framework of Adversarial Filters to investigate model-based reduction of dataset biases. We discuss that the optimum bias reduction via AFOPTIMUM is intractable, thus propose AFLITE, an iterative greedy algorithm that adversarially filters out data points to identify a reduced dataset with more realistic problem distributions and considerably less spurious biases. AFLITE is lightweight and can in principle be applied to any task and dataset. We apply it to popular benchmarks that are practically solved — ImageNet and Natural Language Inference (SNLI, MNLI, QNLI) — and present filtered counterparts as new challenge datasets where the model performance drops considerably (e.g., from 84% to 24% for ImageNet and from 92% to 62% for SNLI), while human performance remains high. An extensive suite of analysis demonstrates that AFLITE effectively reduces measurable dataset biases in both the synthetic and real datasets. Finally, we introduce new measures of dataset biases based on K-nearest-neighbors to help guide future research on dataset developments and bias reduction.
sections.heading: 1 INTRODUCTION
sections.text: Large-scale neural networks have achieved superhuman performance across many popular AI benchmarks, for tasks as diverse as image recognition (ImageNet; Russakovsky et al. (2015)), natural language inference (SNLI; Bowman et al. (2015)), and question answering (SQuAD; Rajpurkar et al. (2016)). Yet these deep models struggle when taken out of these dataset environments and evaluated on adversarial data or problems in the wild (Eykholt et al., 2018; Jia & Liang, 2017). This raises a key question: Does high model performance on today’s benchmark datasets indicate the underlying task is solved, or do those datasets overestimate the true capabilities of current AI systems?
  Answering this question is key because benchmarks serve important roles in the community. Not only do they direct progress on core tasks, they also make it easier to tackle the lofty target tasks such as image recognition in the wild through a more practically-scoped dataset such as ImageNet. However, the closed-world assumption of most existing datasets is subject to significant bias (Torralba & Efros, 2011). Much of the data that is easy to obtain and label isn’t necessarily representative of the task we seek to measure. Thus, if left unchecked, artifacts from data collection (Fouhey et al., 2018) or human labeling (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Geva et al., 2019) can significantly inflate model performance. Though there exist task- and dataset-specific ap-
  proaches for addressing these biases (Goyal et al., 2017; Geirhos et al., 2018), the complex artifacts that emerge from large-scale dataset creation are challenging to exhaustively identify and remove.
  In this paper, we present AFLITE – a computationally efficient dataset reduction algorithm, aimed at systematically reducing spurious artifacts in a dataset. AFLITE is general and can be applied to any task and dataset. Our approach leverages a high capacity model to learn dataset specific biases on a small subset, then uses it to identify and filter artifact-prone instances in the remainder of the dataset to yield a final dataset that is possibly closer to the intended task.
  We first evaluate the effectiveness of our method on synthetic data and show that AFLITE lowers the performance of models relying on annotation artifacts while preserving the performance of models whose representation captures the underlying tasks. In addition, while AFLITE aims to retain the more challenging, confusing instances, our experiments show that it can successfully remove biased instances that are adversarial to the correct representation of the data.
  Finally, we apply the method to several benchmark datasets across various tasks and domains. In language understanding, we apply AFLITE to the SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) datasets for natural language inference, and to QNLI (Wang et al., 2018) for question answering. We show a 30% absolute gap in performance in the current state-of-art methods before and after AFLITE. In computer vision, AFLITE reduces the performance of image classification neural methods on ImageNet (Russakovsky et al., 2015), showing a 49% absolute gap.
sections.heading: 2 DATASET REDUCTION FOR REPRESENTATION-BIAS MINIMIZATION
sections.text: In this section, we introduce AFLITE, a general approach for reducing the scope of bias in datasets. Large datasets run the risk of prioritizing performance on the data-rich head of the distribution, where examples are plentiful, and discounting the tail. Our goal is to minimize the ability of a model to exploit biases in the head of the distribution, while preserving the inherent complexity of the tail.
  Let Φ represent a feature representation, defined over a datasetD = (X,Y ). With AFLITE, we seek a subset S ⊂ D of size |S| ≥ n that is maximally resilient to the features uncovered by Φ. For any identically-distributed train-test split of D, the features extracted by Φ should not generalize to the held-out set. Our approach allows for any choice of feature representation.
  Formalization More formally, let M denote a family of classification models (e.g., logistic regression, SVM, or a particular neural architecture) that can be trained on subsets S of D = (X,Y )
  using features Φ(X). We define the representation bias of Φ in S w.r.tM, denotedR(Φ, S,M), as the best possible out-of-sample classification accuracy achievable by models inM when predicting the true labels Y using features Φ(X). For a given target reduced dataset size of at least n, the goal is to find a subset S ⊂ D, |S| ≥ n that minimizes this representation bias in S w.r.t.M:
  min S⊂D, |S|≥n
  R(Φ, S,M) (1)
  Eq. (1) corresponds to the optimum bias reduction, referred to as AFOPTIMUM. R(Φ, S,M) can be formulated as the expected classification accuracy resulting from the following process. Let q : 2S → [0, 1] be a probability distribution over subsets T = (XT , Y T ) of D. The process is to randomly choose a subset T with probability q(T ), train a bias estimator MT ∈ M on D \ T , and evaluate its classification accuracy fMT (Φ(X
  T ), Y T ) on T . Note that the resulting classification accuracy on T itself is a random variable, since the training set D \ T is random. We define the expected value of this classification accuracy to be the representation bias:
  R(Φ, S,M) , ET∼q [ fMT (Φ(X T ), Y T ) ]
  (2)
  While this expression formalizes the intended objective function, it involves a large summation over subsets T ⊂ S just to compute the representation bias present in a single set S. It does not suggest a practical way to compute the minimization in Eq. (1) without further considering each of the exponentially many subsets S ⊂ D individually – thus an optimal solution for Equation (2) is intractable. To get around this difficulty, we reformulate the representation bias in S as a sum factored over the |S| individual instances i ∈ S. This will allow us to efficiently decide whether or not to include i in the targeted, reduced subset we are constructing.
  The idea is to aggregate the contribution of each i towards the representation bias expression across all random choices of the training set D \ T . We call this the predictability score p(i) for i: on average, how reliably can the label yi be predicted using features Φ(xi) when a model fromM is trained on a randomly chosen training set D \ T not containing i. The higher the value of p(i), the easier it is to correctly classify the instance (xi, yi) using model familyM. This is the signal we will use to decide whether to include i in the reduced subset S we are constructing.
  With some abuse of notation, for i ∈ D, let q(i) , ∑
  T3i q(T ) denote the marginal probability of choosing a subset T that contains i. The ratio q(T )q(i) is then the probability of T conditioned on it containing i. Let fMT (Φ(xi), yi) be the classification accuracy of MT on i. The reformulation of representation bias in terms of predictability scores of individual instances works as follows:
  ET∼q [ fMT (Φ(X T ), Y T ) ] = ∑ T⊂S q(T ) · 1|T | ∑ i∈T fMT (Φ(xi), yi)
  = ∑ T⊂S ∑ i∈T q(T ) · fMT (Φ(xi), yi)|T |
  = ∑ i∈S ∑ T⊂S T3i q(T ) · fMT (Φ(xi), yi)|T |
  = ∑ i∈S q(i) ∑ T⊂S T3i q(T ) q(i) fMT (Φ(xi), yi) |T |
  = ∑ i∈S q(i)ET⊂S, T3i [ fMT (Φ(xi), yi) |T | ] = ∑ i∈S p(i)
  where p(i) is the predictability score of i defined as: p(i) , q(i)ET⊂S, T3i [ fMT (Φ(xi), yi)
  |T |
  ] (3)
  While the method works for any probability distribution q with non-zero support on all samples, for simplicity of exposition, we restrict q to be the uniform distribution over all subsets T ⊂ S of a fixed size. This makes both |T | and q(i) fixed constants; in particular, q(i) = (|S|−1 |T |−1 ) / (|S| |T | )
  = |T ||S| . This reduces the predictability score expression of Eq. (3) to the simplified variant p̃(i):
  p̃(i) , 1
  |S| ET⊂S, T3i [fMT (Φ(xi), yi)] (4)
  Putting the pieces together, we have a factored reformulation of the representation bias in Eq. (2): R(Φ, S,M) = ∑ i∈S p̃(i) (5)
  Armed with this factored representation, we return to the task of identifying an S ⊂ D, |S| ≥ n that minimizes the representation bias. We use the simplified predictability scores (henceforth simply referred to as predictability scores) as a heuristic metric to decide which i ∈ D to include in S. We consider three approaches that iteratively filter out the most predictable instances fromD to arrive at S. In all cases, we use a fixed training set size |S \ T | = t < n. Further, since a larger filtered set is generally desirable, we terminate the filtering process early (i.e., while |S| > n) if the predictability score for every i falls below a pre-specified early stopping threshold τ ∈ [0, 1]. The three approaches are as follows. (A) A simple greedy approach starts with the full set S = D, identifies an i ∈ S with the highest predictability score, removes it from S, and repeats up to |D|−n times. (B) A greedy slicing approach identifies the instances with the k highest predictability scores, removes all of them from S, and repeats the process up to b |D|−nk c times. (C) A slice sampling approach where, instead of greedily choosing the top k instances, it randomly samples k instances with probabilities proportional to their predictability scores.1 In this paper, we use the greedy slicing approach in our experiments and refer to it as AFLITE. While the optimum bias reduction via AFOPTIMUM is intractable, its light-weight version, AFLITE, is applicable in practice.
  The slice sampling approach can be efficiently implemented using what is known as the Gumbel method or Gumbel trick (Gumbel & Lieblein, 1954; Maddison et al., 2014), which uses random perturbations to turn sampling into a simpler problem of optimization. This has recently found success in several probabilistic inference applications (Kim et al., 2016; Jang et al., 2016; Maddison et al., 2016; Balog et al., 2017; Kool et al., 2019). Starting with the log-predictability scores log p̃(i) for various i, the idea is to perturb them by adding an independent random noise γi drawn from the standard Gumbel distribution. Interestingly, the maximizer i∗ of γi+log p̃(i) turns out to be an exact sample drawn from the (unnormalized) distribution defined by p̃. Note that i∗ is a random variable since the γi are drawn at random. This result can be generalized (Vieira, 2014) for slice sampling: the k highest values of Gumbel-perturbed log-predictability scores correspond to sampling, without replacement, k items from the probability distribution defined by p̃. The Gumbel method is typically applied to exponentially large combinatorial spaces, where it is challenging to scale up. In our setting, however, the overhead is minimal since the cost of drawing a random γi is negligible compared to computing p̃(i).
  Implementation Algorithm 1 provides an implementation of AFLITE. The algorithm takes as input a dataset D = (X,Y ), a representation Φ(X) we are interested in minimizing the bias in, a model family M (e.g., linear classifiers), a target dataset size n, size m of the support of the expectation in Eq. (4), training set size t for the classifiers, size k of each slice, and an early-stopping filtering threshold τ . Importantly, for efficiency, Φ(X) is provided to AFLITE in the form of precomputed embeddings for all of X . To obtain Φ(X) in practice, we train a first model on a small fraction of the data based on the learning curve in low-data regime, and do not reuse this data for the rest of our experiments. Moreover, this fraction corresponds to the training size t for AFLITE and it remains unchanged across iterations. We follow the iterative filtering approach, starting with S = D and iteratively removing some instances with the highest predictability scores using the
  1All of these approaches can be improved further by considering not only the predictability score of the top-k instances i but also (via retraining without these instances) how their removal would influence the predictability scores of other instances in the next step. We found our computationally lighter approaches to work well even without the additional overhead of such look-ahead.
  Algorithm 1: AFLITE Input: dataset D = (X,Y ), pre-computed representation Φ(X), model familyM, target dataset size n, number of random partitions m, training set size t < n, slice size k ≤ n, early-stopping threshold τ Output: reduced dataset S
  1 S = D 2 while |S| > n do // Filtering phase 3 forall i ∈ S do 4 Initialize a multi-set of out-of-sample predictions E(i) = ∅ 5 for iteration j : 1..m do 6 Randomly partition S into (Tj , S \ Tj) s.t. |S \ Tj | = t 7 Train a classifier L ∈M on {(Φ(x), y) | (x, y) ∈ S \ Tj} (L is typically a linear classifier) 8 forall i = (x, y) ∈ Tj do 9 Add the prediction L(Φ(x)) to E(i)
  10 forall i = (x, y) ∈ S do 11 Compute the predictability score p̃(i) = |{ŷ ∈ E(i) s.t. ŷ = y}| / |E(i)| 12 Select up to k instances S′ in S with the highest predictability scores subject to p̃(i) ≥ τ 13 S = S \ S′ 14 if |S′| < k then 15 break 16 return S
  greedy slicing strategy. Slice size k and number of partitions m are determined by the available computation budget.
  At each filtering phase, we train models (linear classifiers in our implementation) on m different random partitions of the data, and collect their predictions on their corresponding test set. For each instance i, we compute its predictability score as the ratio of the number of times its label yi is predicted correctly, over the total number of predictions for it. We rank the instances according to their predictability score and use the greedy slicing strategy of removing the top-k instances whose score is not less than the early-stopping threshold τ . We repeat this process until fewer than k instances pass the τ threshold in a filtering phase or fewer than n instances remain.
sections.heading: 3 EXPERIMENTAL ANALYSIS
sections.text: We evaluate AFLITE across various domains (synthetic, natural language processing, computer vision), different tasks in a given domain (language inference and question answering in NLP), different datasets for a given task (SNLI and MNLI in natural language inference), and different representations for a given dataset (pre-computed embeddings from ESIM+GLoVe, BERT, RoBERTa for the SNLI dataset).
sections.heading: 3.1 SYNTHETIC EXPERIMENTS
sections.text: We demonstrate the utility of AFLITE in a synthetic data setting. Our dataset consists of twodimensional data, arranged in concentric circles, at four different levels of separation, as shown in the Figure 2. As is evident, a linear function might not be adequate for separating the two classes; it requires a more complex non-linear model such as an SVM with an RBF kernel. 2
  We add class-specific artificially constructed features (artifacts) sampled from two different Gaussian distributions. These features are only added to 75% of the data in each class, while for the rest of the data, we insert random (noise) features. These artifacts make the task solvable through a linear function. Furthermore, for the first dataset, with the largest separation, we flipped the labels of some examples with artifacts, making the data slightly adversarial even to the RBF. Both models can clearly leverage the artifacts, and demonstrate improved performance over a baseline without artifacts.
  2We use standard implementations from scikit-learn: https://scikit-learn.org/stable/.
  Once we apply AFLITE, as expected, the number of examples with artifacts is reduced considerably, making the task hard once again for the linear model, but still solvable for the non-linear one. The filtered dataset is shown in the bottom half of Fig. 2, and the captions indicate the performance of a linear and an SVM model. For the first dataset, we see that AFLITE removes most of those examples with flipped labels.
sections.heading: 3.2 NLP EXPERIMENTS
sections.text: We evaluate AFLITE on two NLP tasks, namely NLI and question answer sentence selection. We use two popular NLI large-scale datasets – SNLI (Bowman et al., 2015) and MNLI (Wang et al., 2018). For the answer sentence selection task, we use QNLI which is a transformed version of the SQuAD question answering dataset (Rajpurkar et al., 2016) converted to binary classification where systems determine whether a sentence contains the answer to a question.
  SNLI Each instance in the SNLI dataset consists of a premise-hypothesis pair that belongs to one out of three possible categories (entailment, contradiction, or neutral) based on the relationship between the premise and the hypothesis.
  For SNLI, we experiment with three different feature representations derived from strong baseline models: ΦBERT and ΦRoBERTa which are based on BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b), large-scale pretrained masked language models, plus ΦESIM+GLoVe which uses the
  ESIM model (Chen et al., 2016) with GLoVe word embeddings (Pennington et al., 2014). In all cases, feature representation Φ is trained on a random sample of 10% of the original training instances, and feature representations are extracted from the final layer before the output layer. These features are pre-computed for all remaining instances while we discard the instances (10% of training) used for training the embeddings in the subsequent steps of our algorithm. Additionally, to measure the ability of a weaker adversary to filter biases only learned by a stronger model, we evaluate the filtered datasets (for SNLI) with three different models: (i) ESIM with ELMo embeddings (Peters et al., 2018), (ii) BERT-large, and (iii) RoBERTa-large models.
  Table 1 shows the results for SNLI. In all cases, applying AFLITE substantially reduces overall model accuracy, with typical drops of 15-35% depending on the models used for learning the feature representations and those used for evaluation of the filtered dataset. In general, performance is lowest when using the strongest model (RoBERTa) for learning feature representations. Results also highlight the ability of weaker adversaries to produce datasets that are still challenging for much stronger models with a drop of 13.7% for RoBERTa using ΦESIM+GLoVe as feature representation. We also include a model that uses Point-wise Mutual Information (PMI) between words in a given instance and the target label as a feature. The baseline captures the extent to which datasets exhibit word-association artifacts. While this baseline is relatively weaker than other models, we still show that its performance reduce from 54.5% on D to 41.9% on the D(φRoBERTa) dataset.
  It might seem unsurprising that reducing the size of the training set results in lower performance. To control for the confounding factor of the dataset size, we create another filtered dataset D92k, sampled randomly from D such that its size is approximately equal to the size of D(φRoBERTa) dataset. All models achieve nearly the same performance as their performance on the full dataset – even when trained on just one-fifth the original dataset size. This result further points to the fact that current benchmark datasets contain significant redundancy within its instances.
  Finally, to demonstrate the value of the iterative, ensemble-based AFLITE algorithm, we compare with a baseline where using a single model, we filter out the most predictable examples in a single iteration — a non-iterative, single-model version of AFLITE. A RoBERTa-large model trained on this subset (of the same size as D(φRoBERTa)) achieves a dev accuracy of 72.1%. Compared to the performance of RoBERTa on D(φRoBERTa) (62.6%, see Table 1), it makes this baseline a sensible yet less effective approach. In particular, this illustrates the need for an iterative procedure involving models trained on multiple partitions of the remaining data in each iteration.
  We also report the k-nearest neighbors distances between examples in the train and heldout data in Table 3. We consider distances for examples within each class, as well as examples across classes. The distances are computed using cosine similarity between pooled features from BERT-based model (features for the [CLS] token, indicating a sentence-pair feature) trained on the original SNLI dataset. Distances are measured between samples from the heldout data, and their nearest neighbors in the training data, before and after filtering. Distances generally increase after filtering, indicating that AFLITE promotes selecting a diverse set of examples from the dataset. The only exception to the rule is the neutral class, where distances to other classes decrease – this is not surprising since the neutral class is known to be associated with the least number of artifacts (Gururangan et al., 2018).
  MNLI and QNLI Following the same procedure described above, we apply AFLITE on the MNLI and QNLI datasets. Since RoBERTa resulted in the largest drops in performance across the board in SNLI, we only experiment with RoBERTa as adversary for MNLI and QNLI. While RoBERTa achieves over 90% on both original datasets, its performance drops to 66.2% for MNLI and to 77.7% for QNLI on the reduced datasets. Similarly, partial input baseline performance also decreases substantially on both dataset compared to their performance on the original dataset. Table 2 shows these results. We show that AFLITE consistently result in reduced accuracy on the filtered datasets across multiple NLP benchmark datasets, even after controlling for the size of the training set.
sections.heading: 3.2.1 OUT-OF-DISTRIBUTION NLI
sections.text: We measure the performance of AFLITE on three other benchmarks for NLI evaluation, which provide out-of-distribution examples to challenge reliance on dataset biases in the original SNLI data (Glockner et al., 2018; Naik et al., 2018). Such benchmarks approximate the performance of NLI models in the wild. NLI Diagnostics (Wang et al., 2018) is a set of hand-crafted examples designed to demonstrate model performance on several fine-grained semantic categories, such as logical reasoning and commonsense knowledge. HANS (McCoy et al., 2019) contains evaluation examples designed to avoid common structural heuristics (such as word overlap) which could be used by models to correctly predict NLI inputs, without true inferential reasoning. Adversarial NLI (Nie et al., 2019) consists of premises collected from Wikipedia and other news corpora, and human generated hypotheses, arranged at different tiers of the challenge they present to a model, using a human and model in-the-loop procedure. Given that these benchmarks are collected independently of the original SNLI task, the biases from SNLI are less likely to carry over; however these benchmarks might contain their own biases (Liu et al., 2019a).
  AFLITE assigns a predictability score to all samples in a dataset, resulting in an ordering of the data. Filtering out examples from the head of the data distribution based on this order yields more accurate benchmarks for measuring true model performance. On the other hand, transferability to out-of-distribution data would involve a greater balance between examples from the head and the tail ends of the data distribution. Hence we evaluate AFLITE for generalization using a larger filtered subset, amounting to only a third of the full training data. We present these results on all the above benchmarks in Table 4. On the two diagnostic datasets (HANS and NLI-Diagnostics), we perform a zero-shot evaluation of the two models. Adversarial NLI allows to test for transfer
  capabilities, by finetuning these models on each of the three training datasets (Rd1, Rd2 and Rd3). On each of the benchmarks above, the model trained on the AFLITE data consistently outperforms the model trained on the full SNLI data. challenging examples in the HANS benchmark, which targets models purely relying on lexical and syntactic cues. Similarly, our model performs better on the instances in NLI-Diagnostics that require logical reasoning and commonsense knowledge, as opposed to instances that can be solved through lexical entailment alone.
sections.heading: 3.3 IMAGENET EXPERIMENTS
sections.text: We evaluate AFLITE on image classification through ImageNet (ILSVRC2012) classification. On ImageNet, we use the state-of-the-art EfficientNet-B7 model as our core feature extractor Φ (Tan & Le, 2019). The EfficientNet model is learned from scratch on a fixed 20% sample of the ImageNet training set, using AutoAugment data augmentation (Cubuk et al., 2019). We then use the 2560- dimensional features extracted by EfficientNet-B7 as then underlying representation for AFLITE to use to filter the remaining dataset.
  In Table 5, we evaluate the robustness of the filtered dataset by considering ImageNet accuracy across the EfficientNet and ResNet model families (He et al., 2016). When lowering the size of the training set – down to 20% of the original, we find a large drop in performance. The EfficientNet models seem to suffer less – from 84% to 73% on EfficientNet-B7 versus 80.6% to 56.5% on ResNet-152. However, the biggest performance drop comes from training and evaluating on the AFLITE-filtered dataset: the top performer is still EfficientNet-B7, but its accuracy drops to 24.5% top-1. This is despite controlling for dataset size, as well as discrepancy between the training and validation sets.
  Overall, these results suggest that image classification – even within a subset of the closed world of ImageNet – is far from solved. These results echo other findings that suggest that common biases that naturally occur in web-scale image data, such as towards canonical poses (Alcorn et al., 2019) or towards texture rather than shape (Geirhos et al., 2018), are problems for ImageNet-trained classifiers. Indeed, the randomly-selected ImageNet images in Figure 1 suggest that the AFLITE algorithm learns to identify subsets of the data that are particularly challenging.
sections.heading: 4 RELATED WORK
sections.text: Our proposed framework for artifact reduction is related to the adversarial filtering (AF) algorithm in Zellers et al. (2018), yet distinct in two key ways: our approach is (i) much more broadly applicable (by not requiring over generation of data instances), and (ii) considerably more lightweight (by not requiring re-training a model at each iteration of AF). Variants of this AF approach have recently been used to create other datasets such as HellaSwag (Zellers et al., 2019) and ANLI (Bhagavatula
  et al., 2019) by iteratively perturbing dataset instances until a target model cannot fit the resulting dataset. While effective, these approaches run into three main pitfalls. First, dataset curators need to explicitly devise a strategy of collecting or generating perturbations of a given instance. Second, the approach runs the risk of distributional bias where a discriminator can learn to distinguish between machine generated instances and human-generated ones. Finally it requires re-training a model at each iteration, which is computationally expensive especially when using a large model such as BERT (Devlin et al., 2019) as the adversary. In contrast, AFLITE focuses on addressing dataset biases from existing datasets instead of adversarially perturbing instances. AFLITE was earlier proposed by Sakaguchi et al. (2019) to create the Winogrande dataset. This paper presents more thorough experiments, theoretical justification and results from generalizing the proposed approach to multiple popular NLP and Vision datasets.
  AFLITE is also inspired by Gururangan et al. (2018), who study lexical biased prevalent in the SNLI dataset (Bowman et al., 2015) and use point-wise mutual information (PMI) between a word and an inference class to determine the words that are highly indicative of the target label. Instead of lexical features, we adopt a deeper representation of the instances using their pre-computed dense feature representations. We use an ensemble of linear classifiers trained on random subsets of the data to determine whether the dense feature representations are highly indicative of the target label. If so, we discard the corresponding instances and proceed iteratively.
  Li & Vasconcelos (2019) recently proposed REPAIR, a method to remove representation bias by dataset resampling. While resampling is a common technique for balancing datasets, the motivation in REPAIR is to learn a probability distribution over the dataset that favors instances that are hard for a given representation. This approach targets how to train better, less-biased models as opposed to creating datasets with fewer artifacts. In addition, the implementation of REPAIR relies on intraining classification loss as opposed to out-of-sample generalization accuracy. RESOUND (Li et al., 2018) is another method that quantifies the representation biases of datasets. It uses the representation biases to assemble a new K-class dataset with smaller biases by sampling an existing C-class dataset (C > K).
  Arjovsky et al. (2019) argue that unstable, spurious correlations in the data would generalize poorly to novel test environments. Thus, they propose Invariant Risk Minimization as an objective that promotes learning representations of the data which are stable across environments. Instead of learning optimal classifiers, our aim is to remove instances that exhibit artifacts in a dataset.
sections.heading: 5 CONCLUSION
sections.text: We presented AFLITE – a novel iterative greedy algorithm that adversarially filters out data points to arrive at a reduced dataset with more realistic problem distributions and considerably fewer spurious biases. We apply AFLITE to four widely-used datasets, including SNLI and ImageNet, where reported performance is extremely high – and show that state-of-the-art performance on the resulting filtered dataset drops by 30 points for SNLI and drops from 84.4% to 24.5% Top-1 accuracy for ImageNet. In extensive analysis we show that AFLITE is effective on real as well as synthetic datasets. We hope that dataset creators will employ AFLITE to identify unobservable artifacts before releasing new challenge datasets for the research community in order to have a more reliable estimate of model performance on future AI benchmarks.