Recent work (Feng et al., 2018) establishes the presence of short, uninterpretable input fragments that yield high confidence and accuracy in neural models. We refer to these as Minimal Prediction Preserving Inputs (MPPIs). In the context of question answering, we investigate competing hypotheses for the existence of MPPIs, including poor posterior calibration of neural models, lack of pretraining, and dataset bias" (where a model learns to attend to spurious, non-generalizable cues in the training data). We discover a perplexing invariance of MPPIs to random training seed, model architecture, pretraining, and training domain. MPPIs demonstrate remarkable transferability across domains  achieving significantly higher performance than comparably short queries. Additionally, penalizing over-confidence on MPPIs fails to improve either generalization or adversarial robustness. These results suggest the interpretability of MPPIs is insufficient to characterize generalization capacity of these models. We hope this focused investigation encourages more systematic analysis of model behavior outside of the human interpretable distribution of examples. 1 Introduction Feng et al. (2018) establish the presence of shortened input sequences that yield high confidence and accuracy for non-pretrained neural models. These Minimal Prediction Preserving Inputs (MPPIs) are constructed by iteratively removing the least important word from the query to obtain the shortest sequence for which the model's prediction remains unchanged (example shown in Figure 1). 1 Humans are unable to make either confident or accurate predictions on these inputs. Follow up work treats  equal contribution 1 For question answering we construct MPPIs by only removing words from the query. Modifying the context paragraph is poorly defined in MPPI generation as it perturbs the output space, rendering an answer impossible or trivial. SQUAD Context ... The site currently houses three cinemas, including the restored Classic the United Kingdom's last surviving news cinema still in full-time operationalongside two new screens ... Original What's the name of the United Kingdom 's sole remaining news cinema ? Reduced news Confidence 0.57  0.51 Figure 1: A SQUAD dev set example. Given the original Context , the model makes the same correct prediction (Classic) on the Reduced question (MPPI) as the Original , with almost the same score. For humans, the reduced question, news, is nonsensical. strong model performance on such partial-inputs as equivalent with models improperly learning the task (Feng et al., 2019; Kaushik and Lipton, 2018; He et al., 2019). Accordingly, we evaluate this proposition in question answering (QA), investigating the properties of MPPIs and how their existence relates to dataset bias", out-of-domain generalization, and adversarial robustness. First we examine the hypothesis that MPPIs are a symptom of poor neural calibration. Feng et al. (2018) propose we can attribute [these neural] pathologies primarily to the lack of accurate uncertainty estimates in neural models. As neural models tend to overfit the log-likelihood objective by predicting low-entropy distributions (Guo et al., 2017) this can manifest in over-confidence on gibberish examples outside of the training distribution (Goodfellow et al., 2014). We test this hypothesis using pretrained models, shown to have better posterior calibration and out-of-distribution robustness (Hendrycks et al., 2020; Desai and Durrett, 2020). Contrary to expectations, we find large-scale pretraining does not produce more human interpretable MPPIs. flawed annotation procedure results in hidden linguistic cues or annotation artifacts" (Gururangan et al., 2018; Niven and Kao, 2019). Models trained on such data distribution can rely on simple heuristics rather than learning the task. As such, input fragments or partial inputs" are often sufficient for a model to achieve strong performance on flawed datasets. This explanation has been considered for both Natural Language Inference tasks (the hypothesis-only" input for Poliak et al. (2018); Gururangan et al. (2018)) and for Visual Question Answering (the question-only" model for Goyal et al. (2017)). We expect models which rely on these spurious cues would fail to generalize well to other domains" (datasets with different collection and annotation procedures). We discover even models trained in different domains perform nearly as well on MPPIs as on full inputs, contradicting this hypothesis. Further, we test their transferability across a number of other factors, including random training seed, model size, and pretraining strategy, and confirm their invariance to each of these. Third we examine the hypothesis that MPPIs inhibit generalization. This intuition is based on MPPI's poor human interpretability, which could suggest models should not attend to these signals. To test this hypothesis, we regularize this phenomenon directly to promote more human understandable MPPIs, and measure the impact on out-domain generalization and adversarial robustness. Interestingly, out-domain generalization and robustness on Adversarial SQUAD (Jia and Liang, 2017) vary significantly by domain, with both declining slightly on average due to regularization. In conjunction, these results suggest MPPIs may represent an unique phenomenon from what previous work has observed and analyzed. The performance of these inputs is not well explained by domain-specific biases, or posterior over-confidence on out-of-distribution inputs. Instead, this behavior may correspond to relevant signals as the impact of their partial mitigation suggests. We hope these results encourage researchers to not assume MPPIs, or other uninterpretable model behaviour, are dataset artifacts that require mitigation a priori. Before presenting mitigation solutions, we propose they follow a more systematic analysis proposed by our actionable framework by (a) rigorously testing the alleged causes of the observed behaviour, (b) confirming the bias does not general-ize/transfer, and (c) ensuring the solution provides Dataset ORIGINALBERT-B XLNET-LSQUAD (Rajpurkar et al., 2016) 11.54 2.32 2.65 HOTPOTQA (Yang et al., 2018) 18.96 2.07 2.55 NEWSQA (Trischler et al., 2016) 7.59 2.08 1.80 NATURALQ (Kwiatkowski et al., 2019) 9.17 1.22 1.26 TRIVIAQA (Joshi et al., 2017) 15.68 2.33 1.80 SEARCHQA (Dunn et al., 2017) 17.43 1.81 1.05 Table 1: Number of MPPI query tokens, for different datasets and models. All models trained, including DRQA (Chen et al., 2017), BERT (Devlin et al., 2019), and XLNET (Yang et al., 2019), employ setup and parameter choices from Longpre et al. (2019). 2 We generate MPPIs by iteratively removing the least important word from the question, while keeping the original prediction unchanged. The least important word is given as that for which the model's confidence in its prediction remains highest in its absence. 3 To examine how MPPIs transfer across Question Answering domains we employ 6 diverse QA training sets and 12 evaluation sets. 4 The datasets were selected for annotation variety, differing on: question type, document source, annotation instructions, whether the question was collected independently of the passage, and skills required to answer the question. This set represents a realistic spectrum of domains for evaluating generalization. We set aside 2 k examples from each domain's validation sets in order to generate MPPIs for model evaluation. For each experiment we also generate a set of randomly shortened queries to compare against the MPPIs  we refer to this as the Random MPPI" baseline. For each of the original examples, we generate this baseline by randomly removing words until the length matches that of the corresponding MPPI. 3 Experiments 3.1 Invariance of MPPIs Feng et al. (2018) establish the human-insufficiency" property of MPPIs for non-pretrained, LSTM and attention-based models, in-2 For DRQA, we borrowed the hyper-parameters from hitvoice ( https://github.com/hitvoice/DrQA )) 3 Details of model training and examples of MPPI generation are described in Appendix A. 4 Refer to Appendix A.3 for details, or the MRQA 2019 workshop: https://mrqa.github.io/shared . Fisch et al. (2019) normalized these datasets into purely answerable, extractive format. cluding DRQA, and BIMPM (Wang et al., 2017). We extend this investigation for modern, pretrained Transformers, and assess the invariance" of MPPIs: measuring whether they are random, or are affected by model architecture, pretraining strategy, or training dataset (domain). In subsequent experiments we compare sets of MPPIs using the mean Exact Match or Generalized Jaccard Similarity (GJS), a variant of Jaccard Similarity, which accounts for the possibility of repeated tokens in either of the sequences being compared. Generalized Jaccard Similarity is defined between two token sequences X and Y in Equation 1. Here, n is the index of every element that appears in X  Y . GJS ( X, Y ) = (cid:80) ni =1 min ( X i , Y i ) (cid:80) ni =1 max ( X i , Y i ) (1) We will refer to this as Jaccard Similarity" for simplicity. First, we investigate whether MPPIs are random", or influenced by weight initialization and training data order. Measuring the Jaccard Similarity between MPPI sequences produced by models with different training seeds we find JSMPPI = 57 . 1%  1 . 2 , as compared to JSR = 13 . 8%  0 . 8 on the Random MPPI baseline. This suggests MPPIs are not simply the side-effect of randomness in the training procedure. 3.1.2 Pretraining and Architecture One hypothesis is that traditional LSTM-based models, such as DRQA, do not have sufficient pretraining or world knowledge" to rely on the entire sequence, and overfit to subsets of the input. If this were the primary source of MPPIs, we might expect models that are better calibrated and more robust to out-of-distribution examples to have longer and more interpretable MPPIs. Accordingly, we test this hypothesis with large pretrained transformers, which recent work demonstrates have Train Dataset Reduction Dataset SQUAD HOTPOTQA NEWSQA NATRUALQ SQUAD (-) 31.4 (8.8) 41.0 (21.6) 29.2 (12.5) HOTPOTQA 39.7 (12.8) (-) 39.6 (18.8) 33.8 (13.5) NEWSQA 41.1 (13.0) 31.6 (7.2) (-) 35.2 (12.5) NATRUALQ 37.5 (12.7) 28.7 (7.1) 40.2 (17.9) (-) Average 39.4 (12.8) 30.6 (7.7) 40.3 (19.4) 32.7 (12.8) Table 3: The Jaccard Similarity (%) between BERT generated MPPIs, across domains. Specifically, Desai and Durrett (2020) examine 3 separate NLP tasks, using challenging out-of-domain settings, where models face more examples they should be uncertain about", and find that when used out-of-the-box, pretrained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as 3.5 lower". Similarly Hendrycks et al. (2020) systematically show Pretrained transformers are also more effective at detecting anomalous or [out-of-distribution] examples". These findings suggest pretrained transformers should produce more interpretable MPPIs than non-pretrained models. However, in Table 1 we show MPPIs remain incomprehensibly short for all 6 domains and even for pretrained transformer models (DRQA produces MPPIs on SQUAD of mean length 2 . 04 ). In Table 2 we show MPPIs produced by different model architectures and pretraining strategies are similar, significantly exceeding the Jaccard Similarity of the Random MPPI baseline (JSR = 13 . 8% ). This would not be problematic if pretrained models produced lower confidences for MPPIs than the original examples (demonstrating some form of calibration). However, we find the opposite is true. Taking SQuAD for instance we see in 85% of cases the BERT model is more confident on the MPPI than the original example. Lastly, we verify with manual grading tasks that the MPPIs for BERT and XLNet are no more interpretable to humans than DrQA's MPPIs, as shown in Table 5. This suggests that short, uninterpretable MPPIs are ubiquitous in modern neural question answering models and unmitigated by large scale pretraining, or improved out-of-distribution robustness. Next, we investigate the extent to which MPPIs are domain-specific. We do this by measuring their similarity when produced by models trained in different domains. If MPPIs are the product of bias in the training data, such as annotation artifacts, we would expect them to be relatively domain spe-cific, as different datasets carry different biases. In Table 3 a model trained from each domain (Train Dataset) generates MPPIs for each other domain (Reduction Dataset). For each Reduction Dataset, we measure the mean Jaccard Similarity between MPPIs produced by the Train Dataset model and MPPIs produced by the Reduction Dataset (in-domain) model. In parentheses we show the mean Jaccard Similarity between the Random MPPIs and the Train Dataset MPPIs. In all cases, MPPIs demonstrate higher similarity than the random baseline, indicating that they are not domain specific. 3.2 Cross-Domain Transferability of MPPIs Even when models generate different MPPIs, they may still transfer to the other domain. We would like to measure MPPI transferability, independent of their similarity between models. If QA models perform well on MPPIs generated from a range of domains then this would suggest they are not a product of bias in the training data. Instead, they may retain information important to question answering, rather than annotation artifacts. To better measure the extent of MPPI transferability, we (a) train one model on SQuAD (Train Dataset), and another on NewsQA (Reduction Dataset), (b) use the NewsQA-model to generate 2 k MPPIs on the NewsQA evaluation set, and (c) measure the F1 performance of the SQuAD-model evaluated on both the original NewsQA evaluation set and the MPPI queries as generated in part (b). Figure 2 shows performance on out-domain MPPIs are 46 . 6% closer to original performance than on Random MPPIs. This evidence suggests MPPIs are highly transferable across domains. Consequently, MPPIs may relate to generalization, despite their poor human interpretability. Even though MPPIs are highly transferable between domains, their presence may be associated with poor generalization. To evaluate this possibility, we examine whether the penalization of MPPIs improves generalization, or adversarial robustness. While penalizing over-confidence on MPPIs has Train Dataset F1 Score (%) (cid:52) ID OD (cid:52) OD AR (cid:52) AR SQUAD -0.8 52.9 -1.5  2.3 72.1 +3.1 HOTPOTQA +0.6 48.5 -0.6  1.2 45.5 +1.0 NEWSQA -0.9 53.0 -0.9  0.6 62.9 -1.8 NATURALQ +0.9 51.6 -2.9  3.5 54.9 -0.9 TRIVIAQA -0.6 42.3 -4.1  2.8 38.9 -1.1 SEARCHQA -0.5 38.0 -5.9  2.9 32.3 -4.0 OVERALLAVG -0.2 47.7 -2.7  1.1 51.1 -0.6 Table 4: The impact of MPPI regularization on in-domain (ID) performance, macro-average out-domain (OD) generalization over 12 evaluation datasets, and adversarial robustness (AR) on Adversarial SQUAD. been shown to maintain equivalent in-domain performance, and yield subsequently longer and more human interpretable MPPI queries (Feng et al., 2018), its impact on generalization or robustness has not yet been examined. We employ a simplified version of the MPPI penalization used by Feng et al. (2018), training a model with equal quantities of regular and MPPI examples  maintaining normal QA loss terms for the regular examples, and applying an entropy penalty to MPPI examples. 5 When penalizing over-confidence on MPPIs, we confirm the new MPPI length is significantly longer (Appendix sections B), and more human interpretable (Table 5). In Table 4 we show the difference in F1 scores ( (cid:52) ) between the regularized and original models. Results demonstrate that in-domain F1 (ID), macro-average out-domain F1 over 12 datasets (OD), and 5 See Appendix section A.4 for details. adversarial robustness F1 on Adversarial SQUAD (AR) all decline slightly on average with MPPI regularization  by 0 . 2% , 2 . 7% , and 0 . 6% respectively. These results suggest a model's ability to make predictions on MPPIs is not strongly correlated with either generalization or robustness across 13 total QA datasets. However, the relative stability of in-domain performance as compared to out-domain performance suggests mitigating MPPIs is more harmful when crossing domain boundaries. Certain train datasets exhibit greater sensitivity to MPPI regularization than others. For instance SearchQA is drastically affected in all measures, HotpotQA hardly at all, and SQuAD actually improves by 3 . 1% in adversarial robustness. Additionally, Table 4 shows the 95% confidence intervals for out-domain generalization are often as large as the mean change in performance. Empirically, this demonstrates the effect of MPPI regularization is not consistent, having both positive and negative impacts on performance, depending on which of the 12 out-domain datasets is in question. 6 4 Discussion In SQUAD, the most common MPPI is the empty string (40%). Among non-empty strings, the most common MPPI tokens are: what", ?", who", how", when". Despite the pattern of interrogative words, these tokens are already among the most frequent in SQUAD questions, so it's challenging to measure the unique information they convey. A more direct approach to understand the informative signal of MPPIs is to measure their human insufficiency" property directly. We conduct a grading task, comparing human ability to answer real, MPPI, and random MPPI queries. Table 5 shows that humans could only correctly 6 See Figure 9 in Appendix A.4 for details. answer BERT and XLNet MPPIs slightly more often than random MPPIs (32% and 26% exact match compared to 17%), but could answer 43.5% of MPPIs produced by MPPI-regularized BERT. Although this confirms MPPI-regularization partially resolves over-confident behaviour for these human non-interpretable inputs, we've observed the resulting model fares slightly worse in domain generalization and robustness. We find no evidence that MPPIs are explained by poorly calibrated neural models, lack of pretraining knowledge, or dataset-specific bias. Alternatively they may relate, at least in part, to useful and transferable signals. While practitioners, especially in model debiasing tasks, have focused on human understandable and generalizable features, this work would encourage them to also consider the presence of generalizable features which are not human interpretable. This observation closely relates to prior work in computer vision suggesting human uninterpretable, adversarial examples can be the result of features", not bugs", in which Ilyas et al. (2019) observe a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data." We hope this work provides a framework to rigorously evaluate the impact of bias mitigation methods on robustness and generalization, and encourages ML practitioners to examine assumptions regarding unexpected model behaviour on out-of-distribution inputs. We empirically verify the surprising invariance of MPPIs to random seed, model architecture, and pretraining, as well as their wide transferability across domains. These results suggest that MPPIs may not be best explained by poorly calibrated neural estimates of confidence or dataset-specific bias. Examining their relationship to generalization and adversarial robustness, we highlight the ability to maintain in-domain performance but significantly alter out-domain performance and robustness. We hope our results encourage a more systematic analysis of hypotheses regarding model behavior outside the human interpretable distribution of examples. We would like to acknowledge Eric Wallace, Shi Feng, Jordan Boyd-Graber, Christopher Clark, Drew Frank, Kanit Wongsuphasawat, Ni Lao, and Charlie Maalouf for their guiding insights and helpful discussion.
abstain abstain other abstain result result abstain abstain result method method abstain abstain method abstain abstain abstain abstain result objective abstain abstain other abstain abstain abstain abstain abstain abstain abstain other abstain abstain other other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain method abstain method result abstain result result other