Widespread adoption of deep models has motivated a pressing need for approaches to interpret network outputs and to facilitate model debugging. Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction. Influence functions (IF; Koh and Liang 2017) provide machinery for doing this by quantifying the effect that perturbing individual train instances would have on a specific test prediction. However, even approximating the IF is computationally expensive, to the degree that may be prohibitive in many cases. Might simpler approaches (e.g., retrieving train examples most similar to a given test point) perform comparably? In this work, we evaluate the degree to which different potential instance attribution agree with respect to the importance of training samples. We find that simple retrieval methods yield training instances that differ from those identified via gradient-based methods (such as IFs), but that nonetheless exhibit desirable characteristics similar to more complex attribution methods. Code for all methods and experiments in this paper is available at: https://github.com/successar/instance_attributions_NLP . Interpretability methods are intended to help users understand model predictions (Ribeiro et al., 2016; Lundberg and Lee, 2017; Sundararajan et al., 2017; Gilpin et al., 2018). In machine learning broadly and NLP specifically, such methods have focused on feature-based explanations that highlight parts of inputs responsible for' the specific prediction. Feature attribution, however, does not communicate a key basis for model outputs: training data. Recent work has therefore considered methods for  Equal contribution A hilarious romantic comedy  y = 1 <latexit sha1_base64="00mhe76YrMN0t/3b7j+R/G9n/sU=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSJ4KkkV9CIUvXisYD+kDWWz3bRLN5uwOxFK6K/w4kERr/4cb/4bt20O2vpg4PHeDDPzgkQKg6777aysrq1vbBa2its7u3v7pYPDpolTzXiDxTLW7YAaLoXiDRQoeTvRnEaB5K1gdDv1W09cGxGrBxwn3I/oQIlQMIpWeuwOKWbjybXXK5XdijsDWSZeTsqQo94rfXX7MUsjrpBJakzHcxP0M6pRMMknxW5qeELZiA54x1JFI278bHbwhJxapU/CWNtSSGbq74mMRsaMo8B2RhSHZtGbiv95nRTDKz8TKkmRKzZfFKaSYEym35O+0JyhHFtCmRb2VsKGVFOGNqOiDcFbfHmZNKsV77xSvb8o127yOApwDCdwBh5cQg3uoA4NYBDBM7zCm6OdF+fd+Zi3rjj5zBH8gfP5A6oJkFA=</latexit> input prediction Attribution methods e.g.: IF, GD, NN (dot) IF, GD :  interested in nothing more than sucking you in  Important positive examples Important negative examples NN (dot) : if you can get past the taboo subject  REP :  a canny crowd pleaser  IF, GD, REP : eerily accurate depiction of depression. surfacing training examples that were influential for a specific prediction (Koh and Liang, 2017; Yeh et al., 2018; Pezeshkpour et al., 2019; Charpiat et al., 2019; Barshan et al., 2020; Han et al., 2020). While such instance-attribution methods provide an appealing mechanism to identify sources that led to specific predictions (which may reveal potentially problematic training examples), they have not yet been widely adopted, at least in part because even approximating influence functions (Koh and Liang, 2017)arguably the most principled attribution methodcan be prohibitively expensive in terms of compute. Is such complexity necessary to identify important' training points? Or do simpler methods (e.g., attribution scores based on similarity measures between train and test instances) yield comparable results? In this paper, we set out to evaluate and compare instance attribution methods, including relatively simple and efficient approaches (Rajani et al., 2020) in the context of NLP (Figure 1). We design qualitative evaluations intended to probe the following research questions: (1) How correlated are rankings induced by gradient and similarity-based attribution methods (assessing the quality of more efficient approx-imations)? (2) What is the quality of explanations in similarity methods compared to gradient-based ones (clarifying the necessity of adopting more complex methods)? We evaluate instance-based attribution methods on two datasets: binarized version of the Stanford Sentiment Treebank (SST-2; Socher et al. 2013) and the Multi-Genre NLI (MNLI) dataset (Williams et al., 2018). We investigate the correlation of more complex attribution methods with simpler approximations and variants (with and without use of the Hessian). Comparing explanation quality of gradient-based methods against simple similarity retrieval using leave-one-out (Basu et al., 2020) and randomized-test (Hanawa et al., 2021) analyses, we show that simpler methods are fairly competitive. Finally, using the HANS dataset (McCoy et al., 2019), we show the ability of similarity-based methods to surface artifacts in training data. Similarity Based Attribution Consider a text classification task in which we aim to map inputs x i to labels y i  Y . We will denote learned representations of x i by f i (i.e., the representation from the penultimate network layer). To quantify the importance of training point x i on the prediction for target sample x t , we calculate the similarity in embedding space induced by the model. 1 To measure similarity we consider three measures: Euclidean distance, Dot product, and Cosine similarity. Specifically, we define similarity-based attribution scores as: NN EUC = (cid:107) f t  f i (cid:107) 2 , NN COS = cos ( f t , f i ) , and NN DOT = (cid:104) f t , f i (cid:105) . To investigate the effect of fine-tuning on these similarity measures, we also derive rankings based on similarities between untuned sentence-BERT (Reimers et al., 2019) representations. Gradient Based Attribution Influence Functions (IFs) were proposed in the context of neural models by Koh and Liang (2017) to quantify the contribution made by individual training points on specific test predictions. Denoting model parameter estimates by   , the IF approximates the effect that upweighting instance i by a small amount (cid:15) i  would have on the parameter estimates (here H is 1 To be clear, there is no guarantee that similarity reflects influence' at all, but we are interested in the degree to which this simple strategy identifies useful' training points, and whether the ranking implied by this method over train points agrees with rankings according to more complex methods. the Hessian of the loss function with respect to our parameters): d  d(cid:15) i =  H  1     L ( x i , y i ,   ) . This estimate can in turn be used to derive the effect on a specific test point x test :   L ( x test , y test ,   ) T  d  d(cid:15) i . Aside from IFs, we consider three other similar gradient-based variations: (1) RIF = cos ( H  12   L ( x test ) , H  12   L ( x i )) . (2) GD = (cid:104)  L ( x test ) ,   L ( x i ) (cid:105) , and (3) GC = cos (   L ( x test ) ,   L ( x i )) . RIF was proposed by Barshan et al. (2020), while GD and GC by Charpiat et al. (2019). Representer Points (REP; Yeh et al. 2018) introduced to approximate the influence of training points on a test sample by defining a classi-fier as a combination of a feature extractor and a ( L 2 regularized) linear layer:  ( x i ,  ) . Yeh et al. (2018) showed that for such models the output for any target instance x t can be expressed as a linear decomposition of data importance of training instances:  ( x t ,   ) = (cid:80) ni  i f (cid:62) i f t = (cid:80) ni k ( x t , x i ,  i ) , where  i = 1  2  n  L ( x i ,y i , )  ( x i , ) . Datasets To evaluate different attribution methods, we conduct several experiments on sentiment analysis and NLI tasks, following prior work investigating the use of IF specifically for NLP (Han et al., 2020). We adopt a binarized version of the Stanford Sentiment Treebank (SST-2; Socher et al. 2013), and the Multi-Genre NLI (MNLI) dataset (Williams et al., 2018). For fine-tuning on MNLI, we randomly sample 10k training instances. Finally, to evaluate the ability of instance attribution methods to reveal annotation artifacts in NLI, we randomly sampled 1000 instances from the HANS dataset (more details in the Appendix). Models We define models for both tasks on top of BERT (Devlin et al., 2019), tuning hyperparameters on validation data via grid search. Our models achieve 90 . 6% accuracy on SST and 71 . 2% accuracy on MNLI (more details in the Appendix). Computing the IF for BERT Deriving the IF for all parameters  of a BERT-based model requires deriving the corresponding Inverse Hessian. We compute the Inverse Hessian Vector Product (IHVP) H  1   L ( x, y,  ) directly because storing the entire matrix of |  | 2 elements is practically impossible (requiring  12 PB of storage). We approximate the IHVP using the LiSSa algorithm 1 2 3 (Agarwal et al., 2017). This method is still expensive to run and is sensitive to the norm of the IHVP approximation. Therefore, for computational reasons we consider IF with respect to the subset of parameters that correspond to the top five layers [IF (Top-5)], and only the last linear layer [IF (linear)], resulting in a few orders of magnitude faster procedure (the algorithm becomes increasingly unstable as we incorporate additional layers). We also use a large scaling factor to aid convergence. In this section, we first investigate the correlation between different methods. Then, to study the quality of explanations we conduct leave-some-out experiments, and further analyze attribution methods on HANS data. We consider four evaluations (more analyses and experimental details in the Appendix). (1) Calculating the correlation of each pair of attribution methods, assessing whether simple methods induce rankings similar to more complex ones. (2) Removing the most influential samples according to each method, retrain, and then observe the change in the predicted probability for the originally predicted class, with the assumption that more accurate attribution methods will cause more drop. (3) We follow randomized-test from (Hanawa et al., 2021) and measure the ranking correlation of methods for (a) randomly initialized and (b) trained models, under the assumption that high correlation here would suggest less meaningful attribution. recover examples that exhibit lexical overlap when tested on the HANS dataset (McCoy et al., 2019). This extends a prior analysis of IF (Han et al., 2020), considering alternative attribution methods. Attribution Methods' Correlation We calculate the Spearman correlation between scores assigned to training samples by different methods, allowing us to compare their similarities. More specifically, we randomly sample 100 test and 500 training samples from datasets and calculate the average resultant Spearman correlations. We report attribution methods' correlation on SST and MNLI datasets in Figure 2 (a more complete version of these figures is in the Appendix). We make the following observations. (1) Gradient methods w/wo normalization appear similar to each other, e.g., GC is similar to RIF and IF is similar to GD, suggesting that Hessian information may not be necessary to provide meaningful attributions (GD and GC do not use the Hessian). (2) There is a high correlation between IF calculated over the top five layers of BERT and IF over only the last linear layer. (3) There is only a modest correlation between similarity-based rankings and gradient-based methods, suggesting that these do differ in terms of the importance they assign to training instances. We report a proportion of common top examples between IF (Top-5) and IF (Linear) in the Appendix, providing further evidence of the high correlation between these methods. most important training samples for 50 random test samples using different attribution methods. We only consider the linear version of methods in the remainder of the paper. All methods seem effective, compared to random sampling. Perhaps surprisingly, for both tasks at least one of the similarity-based approaches performs comparably or better than gradient-based methods, in the sense that removing the top examples according to similarity yields reductions in the predicted probability (which is what one would intuitively hope). Finally, it seems that the models applying some form of normalization to the gradient (i.e., RIF and GC) perform more consistently. This is consistent with contemporaneous work of Hanawa et al. (2021) which argues that this is a consequence of large gradient magnitudes for some samples dominating when normalization is not used. Upon investigating high influential training samples, we observed that similarity-based approaches seem to yield more diverse top instances compared to gradient-based ones. We also found that normalization in gradient-based methods made a large difference. Generic IF-based ranking tends to be dominated by high loss training examples across test examples, whereas normalization provides more diverse top training examples. Further, proportions of shared top examples between methods is provided in the Appendix, clarifying their similar performance. Randomized-Test We report the Spearman correlation between trained and random models for SST and MNLI data in Table 1. This would ideally be small in magnitude (non-zero values indicate correlation). Curiously, gradient-based methods (IF, REP, GD) exhibit negative correlations on the SST dataset. Overall, these results suggest that gradient-based approaches without gradient normalization may be inferior to alternative methods. The simple NN-DOT method provides the best' performance according to this metric. Artifacts and Attribution Methods To investigate whether attribution methods can correctly identify training samples with specific artifacts responsible for model predictions we follow Han et al. (2020): This entails randomly choosing 10k samples from MNLI and treating neutral and contradiction as a single non-entailment label for model fine-tuning. More specifically, we are interested in target samples that the model mispredicts as entailment because of the lexical overlap artifact (lexical overlap is an artifactual indicator of entailment; McCoy et al. 2019). The average lexical overlap rate for 1000 random samples from the HANS dataset is provided in Table 2. As a baseline, we also apply similarity-based methods on top of sentence-BERT embeddings, which as expected appear very similar to random correlation. One can observe that similarity-based approaches tend to surface instances with higher lexical overlap, compared to gradient-based instance attribution methods. Moreover, gradient-based methods without normalization (IF, GD, and REP) perform similar to selecting samples randomly and based on sentence-BERT representations, suggesting an inability to usefully identify lexical overlap. Computational Complexity The computational complexity of IF-based instance attribution methods constitutes an important practical barrier to their use. This complexity depends on the number of model parameters taken into consideration. As a result, computing IF is effectively infeasible if we consider all model parameters for modern, medium-to-large models such as BERT. If we only consider the parameters of the last linear layercomprising O ( p ) parametersto approximate the IF, the computational bottleneck will be the inverse Hessian which can be approximated with high accuracy in O ( p 2 ) . There are ways to approximate the inverse Hessian more efficiently (Pearlmutter, 1994), though this results in worse performance. Similarity-based measures, on the other hand, can be calculated in O ( p ) . With respect to wall-clock running time, calculating the influence of a single test sample with respect to the parameters comprising the top-5 layers of a BERT-based model for SST classification running on a reasonably modern GPU 2 requires  5 minutes. If we consider the linear variant, this falls to < 0 . 01 seconds. Finally, similarity-based approaches require < 0 . 0001 seconds. Extrapolating these numbers, it requires about 6 days to calculate IF (top-5 Layer) for all 1821 test samples in SST, while it takes only around 0.2 seconds for similarity-based methods. Instance attribution methods constitute a promising approach to better understanding how modern NLP models come to make the predictions that they do (Han et al., 2020; Koh and Liang, 2017). However, approximating IF to quantify the importance of train samples is prohibitively expensive. In this work, we investigated whether alternative, simpler and more efficient methods provide similar instance attribution scores. We demonstrated high correlation between (1) gradient-based methods that consider more parameters [IF and GD (top-5)] and their simpler counterparts [IF and GD (linear)], and (2) methods without Hessian information, i.e., IF vs GD and RIF vs GC. We considered even simpler, similarity-based approaches and compared the importance rankings over training instances induced by these to rankings under gradient-based methods. Through leave-some-out, randomized-test, and artifact detection experiments, we demonstrated that these simple similarity-based methods are surprisingly competitive. This suggests future directions for work on fast and useful instance attribution methods. All code necessary to reproduce the results reported in this paper is available at: https://github.com/successar/ instance_attributions_NLP . Deep neural models have come to dominate research in NLP, and increasingly are deployed in the real world. A problem with such techniques is that they are opaque; it is not easy to know why models make specific predictions. Consequently, modern models may make predictions on the basis of attributes we would rather they not (e.g., demographic categories or artifacts' in data). Instance attributionidentifying training samples that influenced a given predictionprovides a mechanism that might be used to counter these issues. However, the computational expense of existing techniques hinders their adoption in practice. By contrasting these complex approaches against simpler alternative methods for instance attribution, we contribute to a better understanding and characterization of the tradeoffs in instance attribution techniques. This may, in turn, improve the robustness of models in practice, and potentially reduce implicit biases in their predictions. We thank the anonymous NAACL reviewers for their detailed feedback. Further, we also thank Matt Gardner, Sanjay Subramanian, and Dylan Slack for their useful comments. This work was sponsored in part by the Army Research Office (W911NF1810328), and funded in part by the NSF grants #IIS-1756023 and #IIS-2008956. The views expressed are those of the authors and do not reflect the policy of the funding agencies.
abstain abstain abstain abstain abstain method objective other abstain abstain abstain other abstain abstain abstain abstain method method abstain method objective result result method method method method method method abstain method abstain method abstain abstain abstain other method method method method method method abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective objective method objective abstain other abstain abstain abstain abstain abstain abstain abstain other other other other