Recent work in Natural Language Processing has focused on developing approaches that extract faithful explanations, either via identifying the most important tokens in the input (i.e. post-hoc explanations) or by designing inherently faithful models that first select the most important tokens and then use them to predict the correct label (i.e. select-then-predict mod-els). Currently, these approaches are largely evaluated on in-domain settings. Yet, little is known about how post-hoc explanations and inherently faithful models perform in out-of-domain settings. In this paper, we conduct an extensive empirical study that examines: (1) the out-of-domain faithfulness of post-hoc explanations, generated by five feature attribution methods; and (2) the out-of-domain performance of two inherently faithful models over six datasets. Contrary to our expectations, results show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain. We find this misleading and suggest using a random baseline as a yardstick for evaluating post-hoc explanation faithfulness. Our findings also show that select-then predict models demonstrate comparable predictive performance in out-of-domain settings to full-text trained models. 1 1 Introduction An explanation or rationale 2 , typically consists of a subset of the input that contributes more to the prediction. Extracting faithful explanations is important for studying model behavior (Adebayo et al., 2020) and assisting in tasks requiring human decision making, such as clinical text classification (Chakrabarty et al., 2019), misinformation detection (Popat et al., 2018; Mu and Aletras, 2020) and legal text classification (Chalkidis et al., 2019, 1 Code available at: https://github.com/ GChrysostomou/ood_faith 2 We use these terms interchangeably throughout our work. 2021). A faithful explanation is one which accurately represents the reasoning behind a model's prediction (Jacovi and Goldberg, 2020) Two popular methods for extracting explanations are through feature attribution approaches (i.e. posthoc explanation methods) or via inherently faithful classifiers (i.e. select-then-predict models). The first computes the contribution of different parts of the input with respect to a model's prediction (Sundararajan et al., 2017; Ribeiro et al., 2016; Shrikumar et al., 2017). The latter consists of using a rationale extractor to identify the most important parts of the input and a rationale classifier, a model trained using as input only the extractor's rationales (Bastings et al., 2019; Jain et al., 2020; Guerreiro and Martins, 2021). 3 Figure 1 illustrates the two approaches with an example. Currently, these explanation methods have been mostly evaluated on in-domain settings (i.e. the train and test data come from the same distribution). However, when deploying models in real-world applications, inference might be performed on data from a different distribution, i.e. out-of-domain (Desai and Durrett, 2020; Ovadia et al., 2019). This can create implications when extracted explanations (either using post-hoc methods or through select-then-predict models) are used for assisting human decision making. Whilst we are aware of the limitations of current state-of-the-art models in out-of-domain predictive performance (Hendrycks et al., 2020), to the best of our knowledge, how faithful out-of-domain post-hoc explanations are has yet to be explored. Similarly, we are not aware how inherently faithful select-then-predict models generalize in out-of-domain settings. Inspired by this, we conduct an extensive empirical study to examine the faithfulness of five 3 We refer to the rationale generator (i.e. generating a rationale mask) from Bastings et al. (2019) and Jain et al. (2020) as a rationale extractor, to avoid any confusion between these approaches and free-text rationales (Wiegreffe et al., 2021). feature attribution approaches and the generalizability of two select-then-predict models in out-of-domain settings across six dataset pairs. We hypothesize that similar to model predictive performance, post-hoc explanation faithfulness reduces in out-of-domain settings and that select-then-predict performance degrades. Our contributions are as follows:  To the best of our knowledge, we are the first to assess the faithfulness of post-hoc explanations and performance of select-then-predict models in out-of-domain settings. We show that post-hoc explanation sufficiency and comprehensiveness show misleading increases in out-of-domain settings. We argue that they should be evaluated alongside a random baseline as yardstick out-of-domain. We demonstrate that select-then-predict classifiers can be used in out-of-domain settings. They lead to comparable predictive performance to models trained on full-text, whilst offering inherent faithfulness. Given a model M , we are interested in explaining why M predicted  y for a particular instance x  X . An extracted rationale R , should therefore represent as accurately as possible the most important subset of the input ( R  x ) which contributed mostly towards the model's prediction  y . Currently, there are two popular approaches for extracting rationales. The first consists of using feature attribution methods that attribute to the input tokens an importance score (i.e. how important an input token is to a model's M prediction  y ). We can then form a rationale R , by selecting the K most important tokens (independent or contiguous) as indicated by the feature attribution method. The second select-then-predict approach focuses on training inherently faithful classifiers by jointly training two modules, a rationale extractor and a rationale classifier , trained only on rationales produced by the extractor (Lei et al., 2016; Bastings et al., 2019; Treviso and Martins, 2020; Jain et al., 2020; Guerreiro and Martins, 2021). Recent studies have used feature attribution approaches as part of the rationale extractor (Jain et al., 2020; Treviso and Martins, 2020), showing improved classifier predictive performance. Having extracted R , we need to evaluate the quality of the explanation (i.e. how faithful that explanation is for a model's prediction). Typically, post-hoc explanations from feature attribution approaches are evaluated using input erasure (Serrano and Smith, 2019; Atanasova et al., 2020; Madsen et al., 2021). This approach masks segments of the input to observe if the model's prediction changed. DeYoung et al. (2020) proposed measuring the comprehensiveness and sufficiency of rationales as faithfulness metrics. A comprehensive rationale is one which is influential to a model's prediction, while a sufficient rationale that which is adequate for a model's prediction (DeYoung et al., 2020). The term fidelity is also used for jointly referring to comprehensiveness and sufficiency (Carton et al., 2020). Carton et al. (2020) suggested normalizing these metrics using the predictions of the model with a baseline input (i.e. an all zero embedding 6921 vector), to account for baseline model behavior. Select-then-predict models are inherently faithful, as their classification component is trained only on extracted rationales (Jain et al., 2020). A good measure for measuring rationale quality is by evaluating the predictive performance of the classifier trained only on the rationales (Jain et al., 2020; Treviso and Martins, 2020). A higher score entails that the extracted rationales are better when compared to those of a classifier with lower predictive performance. Given model M trained on an end-task, we typically evaluate its out-of-domain predictive performance on a test-set that does not belong to the same distribution as the data it was trained on (Hendrycks et al., 2020). Similarly, the model can also extract explanations R for its out-of-domain predictions. Camburu et al. (2018) studied whether generating explanations for language inference match human annotations (i.e. plausible explanations). They showed that this is challenging in-domain and becomes more challenging in out-of-domain settings. In a similar direction, Rajani et al. (2019) and Kumar and Talukdar (2020) examined model generated explanations in out-of-domain settings and find that explanation plausibility degrades compared to in-domain. Kennedy et al. (2020) proposed a method for detecting model bias towards group identity terms using a post-hoc feature attribution approach. Then, they use them for regularizing models to improve out-of-domain predictive performance. Adebayo et al. (2020) have studied feature attribution approaches for identifying out-of-distribution images. They find that importance allocation in out-of-domain settings is similar to that of an in-domain model and thus cannot be used to detect such images. Feder et al. (2021) finally argued that explanations can lead to errors in out-of-distribution settings, as they may latch onto spurious features from the training distribution. These studies indicate that there is an increasing need for evaluating post-hoc explanation faithfulness and select-then-predict performance in out-of-domain settings. To the best of our knowledge, we are the first to examine these. We employ a pre-trained BERT -base and fine-tune it on in-domain training data. We then extract posthoc rationales for both the in-domain test-set and two out-of-domain test-sets. We compute input importance using five feature scoring methods and a random baseline:  Random (RAND ): Random allocation of token importance. Attention (  ): Token importance corresponding to normalized attention scores (Jain et al., 2020). Scaled Attention (    ): Attention scores  i scaled by their corresponding gradients   i =   y  i (Serrano and Smith, 2019). InputXGrad ( x  x ): Attributes input importance by multiplying the input with its gradient computed with respect to the predicted class, where  x i =   y x i (Kindermans et al., 2016; Atanasova et al., 2020). Integrated Gradients ( IG ): Ranking words by computing the integral of the gradients taken along a straight path from a baseline input (zero embedding vector) to the original input (Sundararajan et al., 2017). DeepLift: Ranking words according to the difference between the activation of each neuron and a reference activation (zero embedding vector) (Shrikumar et al., 2017). HardKuma: An end-to-end trained model, where the rationale extractor uses Hard Ku-maraswamy variables to produce a rationale mask z , which the classifier uses to mask the input (Bastings et al., 2019). Model training takes advantage of reparameterized gradients compared to REINFORCE style training employed by Lei et al. (2016) and has shown improved performance (Guerreiro and Martins, 2021). FRESH: We compute the predictive performance of a classifier trained on rationales extracted with feature attribution metrics (see 6922  3.1) using FRESH, following a similar approach to Jain et al. (2020). We extract rationales from an extractor by (1) selecting the topk most important tokens (TOPK) and (2) selecting the span of length k with the highest overall importance (CONTIGUOUS ). We use BERT -base for the extraction and classification components of FRESH similar to Jain et al. (2020). However, for HardKuma we opt using a biLSTM (Hochreiter and Schmidhuber, 1997) as it provides comparable or improved performance over BERT variants (Guerreiro and Martins, 2021), even after hyperparameter tuning. 4 4 Experimental Setup 4.1 Datasets For evaluating out-of-domain model explanation, we consider the following datasets (see Table 1 and Appendix A for details): SST: Stanford Sentiment Treebank (SST) consists of sentences tagged with sentiment on a 5-point-scale from negative to positive (Socher et al., 2013). We remove sentences with neutral sentiment and label the remaining sentences as negative or positive if they have a score lower or higher than 3 respectively (Jain and Wallace, 2019). IMDB: The Large Movie Reviews Corpus consists of movie reviews labeled either as positive or negative (Maas et al., 2011; Jain and Wallace, 2019). Yelp: Yelp polarity review texts. Similar to Zhang et al. (2015) we construct a binary classification task to predict a polarity label by considering one and two stars as negative, and three and four stars as positive. Amazon Reviews: We form 3-way classification tasks by predicting the sentiment (negative, neutral, positive) of Amazon product reviews across 3 item categories: (1) Digital Music ( AmazDigiMu ); (2) Pantry ( AmazPantry ); and (3) Musical Instruments ( AmazInstr ) (Ni et al., 2019). Post-hoc Explanations: We evaluate post-hoc explanations using: Normalized Sufficiency (NormSuff) measures the degree to which the extracted rationales are adequate for a model to make a prediction (DeYoung et al., 2020). Following Carton et al. (2020), we bind sufficiency between 0 and 1 and use the reverse difference so that higher is better: Suff ( x ,  y, R ) = 1  max (0 , p ( y | x )  p ( y |R )) NormSuff ( x ,  y, R ) = Suff ( x ,  y, R )  Suff ( x ,  y, 0) 1  Suff ( x ,  y, 0) (1) where Suff ( x ,  y, 0) is the sufficiency of a baseline input (zeroed out sequence) and  y the model predicted class using the full text x as input. Normalized Comprehensiveness (Norm-Comp) measures the influence of a rationale to a prediction (DeYoung et al., 2020). Similarly to Carton et al. (2020), we bind this metric between 0 and 1 and normalize it: Comp ( x ,  y, R ) = max (0 , p ( y | x )  p ( y | x \R )) NormComp ( x ,  y, R ) = Comp ( x ,  y, R ) 1  Suff ( x ,  y, 0) (2) To measure sufficiency and comprehensiveness across different explanation lengths we compute the Area Over the Perturbation Curve" (AOPC) following DeYoung et al. (2020). We therefore compute and report the average normalized sufficiency and comprehensiveness scores when keeping (for sufficiency) or masking (for comprehensiveness) the top 2%, 10%, 20% and 50% of tokens extracted by an importance attribution function. 5 5 We also present results for each of these rationale lengths in Appendix F. 6923 Train Test Full-text Normalized Sufficiency Normalized Comprehensiveness F1 Rand     DeepLift x  x IG Rand     DeepLift x  x IG SST SST 90.1 0.38 0.51 0.42 0.42 0.40 0.41 0.19 0.39 0.22 0.25 0.26 0.26 IMDB 84.3 0.31 0.53 0.39 0.32 0.31 0.32 0.23 0.54 0.34 0.27 0.27 0.28 Yelp 87.9 0.32 0.56 0.40 0.35 0.33 0.34 0.21 0.48 0.28 0.24 0.24 0.25 IMDB IMDB 91.1 0.32 0.55 0.46 0.36 0.36 0.36 0.16 0.48 0.31 0.25 0.23 0.24 SST 85.8 0.24 0.35 0.28 0.28 0.27 0.27 0.27 0.46 0.32 0.33 0.33 0.33 Yelp 91.0 0.35 0.48 0.41 0.36 0.36 0.36 0.21 0.45 0.32 0.26 0.26 0.26 Yelp Yelp 96.9 0.23 0.32 0.31 0.29 0.24 0.25 0.12 0.20 0.14 0.16 0.15 0.16 SST 86.8 0.41 0.45 0.43 0.44 0.41 0.41 0.17 0.24 0.18 0.21 0.22 0.22 IMDB 88.6 0.18 0.34 0.32 0.25 0.22 0.22 0.19 0.34 0.29 0.23 0.23 0.24 AmazDigiMu AmazDigiMu 70.6 0.34 0.56 0.34 0.31 0.41 0.39 0.13 0.32 0.14 0.10 0.16 0.17 AmazInstr 61.2 0.29 0.54 0.32 0.31 0.33 0.32 0.19 0.47 0.23 0.19 0.22 0.23 AmazPantry 64.6 0.33 0.55 0.33 0.31 0.37 0.36 0.21 0.46 0.22 0.17 0.23 0.25 AmazPantry AmazPantry 70.2 0.25 0.46 0.36 0.19 0.28 0.27 0.20 0.42 0.31 0.15 0.25 0.25 AmazDigiMu 59.5 0.24 0.47 0.37 0.19 0.27 0.26 0.19 0.41 0.32 0.15 0.23 0.24 AmazInstr 64.5 0.17 0.42 0.30 0.15 0.20 0.20 0.24 0.52 0.40 0.23 0.30 0.30 AmazInstr AmazInstr 71.5 0.16 0.34 0.18 0.21 0.18 0.17 0.26 0.52 0.26 0.29 0.28 0.29 AmazDigiMu 61.3 0.21 0.38 0.21 0.22 0.24 0.22 0.23 0.46 0.20 0.22 0.24 0.25 AmazPantry 68.2 0.22 0.39 0.21 0.23 0.24 0.23 0.27 0.51 0.22 0.25 0.27 0.29 Table 2: AOPC Normalized Sufficiency and Comprehensiveness (higher is better) in-domain and out-of-domain for five feature attribution approaches and a random attribution baseline. We omit from our evaluation the Remove-and-Retrain method (Madsen et al., 2021) as it requires model retraining. Whilst this could be applicable for in-domain experiments where retraining is important, in this work we evaluate explanation faithfulness in zero-shot out-of-domain settings. Select-then-Predict Models: We first train select-then-predict models in-domain and then measure their predictive performance on the in-domain test-set and on two out-of-domain test-sets (Jain et al., 2020; Guerreiro and Martins, 2021). Our out-of-domain evaluation is performed without retraining (zero-shot). Similar to full-text trained models, we expect that predictive performance deteriorates out-of-domain. However, we assume that explanations from a select-then-predict model should generalize better in out-of-domain settings when the predictive performance approaches that of the full-text trained model. We do not conduct human experiments to evaluate explanation faithfulness, since that is only relevant to explanation plausibility (i.e. how intuitive to humans a rationale is (Jacovi and Goldberg, 2020)) and in practice faithfulness and plausibility do not correlate (Atanasova et al., 2020). 5 Results 5.1 Post-hoc Explanation Faithfulness Table 2 presents the normalized comprehensiveness and sufficiency scores for post-hoc explanations on in-domain and out-of-domain test-sets, using five feature attribution methods and a random baseline. For reference, we include the averaged F1 performance across 5 random seeds, of a BERT base model finetuned on the full text and evaluated inand out-of-domain (Full-text F1). 6 In-domain results show that feature attribution performance varies largely across datasets. This is in line with the findings of Atanasova et al. (2020) and Madsen et al. (2021) when masking rationales (i.e. comprehensiveness). We find the only exception to be    , which consistently achieves the highest comprehensiveness and sufficiency scores across all in-domain datasets. For example    evaluated on in-domain AmazDigiMu, results in sufficiency of 0.56 compared to the second best of 0.39 with IG. Contrary to our expectations, results show that post-hoc explanation sufficiency and comprehensiveness are in many cases higher in out-of-domain test-sets compared to in-domain. For example using DeepLift, comprehensiveness for the in-domain test-set in Yelp (0.16) is lower compared to the out-of-domain test-sets (0.21 for SST and 0.23 for IMDB). This is also observed when measuring sufficiency with    , scoring 0.32 when tested in-domain on Yelp and 0.45 for the out-of-domain SST test-set. Apart from increased sufficiency and comprehen-6 We report predictive performance for all models and standard deviations in the Appendix. 6924 siveness scores in out-of-domain post-hoc explanations, we also observe increased scores obtained by our random baseline. In fact, the random baseline outperforms several feature attribution approaches in certain cases in out-of-domain settings . As an example, consider the case where the model has been trained on AmazInstr and tested on AmazPantry. Our random baseline achieves a comprehensiveness score of 0.27 while  , DeepLift, x  x perform similarly or lower (0.22, 0.25 and 0.27 respectively). Similarly, using a model trained on Yelp and tested on SST, the random baseline produces equally sufficient rationales to x  x and IG, with all of them achieving 0.41 normalized sufficiency. A glaring exception to this pattern is    , which consistently outperforms both the random baseline and all other feature attribution approaches in inand out-of-domain settings, suggesting that it produces the more faithful explanations. For example with out-of-domain AmazPantry test data, using a model trained on AmazInstr results in sufficiency scores of 0.39 with    . This is a 0.15 point increase compared to the second best ( x  x with 0.24). We recommend considering a feature attribution for producing faithful explanations out-of-domain, if it only scores above a baseline random attribution. We suggest that the higher the deviation from the random baseline, the more faithful an explanation is . 5.2 Select-then-predict Model Performance HardKuma: Table 3 presents the F1-macro performance of HardKuma models (Bastings et al., 2019) and the average rationale lengths (the ratio of the selected tokens compared to the length of the entire sequence) selected by the model. For reference, we also include the predictive performance of a full-text trained biLSTM . Results are averaged across 5 runs including standard deviations in brackets. As expected, predictive performance of HardKuma models degrades when evaluated on out-of-domain data. Surprisingly, though, we find that their performance is not significantly different (t-test; p-value > 0.05) to that of the full-text LSTM in 9 out of the 12 out-of-domain dataset pairs. For example, by evaluating the out-of-domain performance of a HardKuma model trained on AmazDigiMu on the AmazPantry test-set, we record on average a score of 54.3 F1 compared to Train Test Full-text HardKuma L F1 F1 (%) SST SST 81.7 77.6 56.8 IMDB 71.9 65.7 39.5 Yelp 68.7 67.7 32.7 IMDB IMDB 87.4 82.0 1.9 SST 77.5 73.6 16.8 Yelp 41.0 47.2 3.1 Yelp Yelp 96.0 92.4 7.4 SST 80.4 72.4 14.1 IMDB 84.5 73.3 4.7 AmazDigiMu AmazDigiMu 67.6 66.8 18.4 AmazInstr 54.2 53.3 25.8 AmazPantry 55.3 54.7 27.8 AmazPantry AmazPantry 67.9 66.6 18.9 AmazDigiMu 50.9 51.0 11.2 AmazInstr 55.9 57.4 18.2 AmazInstr AmazInstr 67.2 66.7 19.2 AmazDigiMu 54.3 53.7 13.9 AmazPantry 61.1 59.5 24.4 Table 3: F1 macro performance (five runs) for HardKuma models and the selected rationale length (L). Bold denotes no significant difference between HardKuma and Full-text (t-test; p > 0.05). For clarity, we include F1 scores with standard deviations in Appendix C. 55.3 with an LSTM classifier trained on full text. We also observe that HardKuma models trained on SST and IMDB generalize comparably to models trained on full-text when evaluated on Yelp, however the opposite does not apply. Our assumption is that HardKuma models trained on Yelp, learn more domain-specific information due to the large training corpus (when compared to training on IMDB and SST) so they fail to generalize well out-of-domain. Results also show, that the length of rationales selected by HardKuma models depend on the source domain, i.e. training HardKuma on a dataset which favors shorter rationales, leads to also selecting shorter rationales out-of-domain . For example, in-domain test-set explanation lengths are on average 56.8% of the full-text input length for SST. In comparison, training a model on Yelp and evaluating on SST results in rationale lengths of 14.1%. We observe that in certain cases, HardKuma models maintain the number of words, not the ratio to the sequence in out-of-domain settings. For example, in-domain Yelp test-set rationales are about 11 tokens long that is the similar to the length selected when evaluating on IMDB using a model trained on Yelp. This is also observed where in-domain AmazInstr test-set rationales are on average 5 tokens long, which 6925 Train Test Full-text     DeepLift x  x IG SST (20%) SST 90.1 87.7 81.1 84.4 76.3 76.8 IMDB 84.3 81.8 52.6 64.0 55.0 56.3 Yelp 87.9 88.1 72.6 75.4 59.6 63.9 IMDB (2%) IMDB 91.1 87.9 80.4 87.2 59.8 59.7 SST 85.8 80.9 71.8 70.1 69.6 70.7 Yelp 91.0 87.8 82.0 79.4 69.0 69.1 Yelp (10%) Yelp 96.9 94.0 90.4 93.6 70.5 71.9 SST 86.8 59.3 69.8 67.2 67.7 69.3 IMDB 88.6 78.0 64.5 66.6 53.0 55.8 AmazDigiMu (20%) AmazDigiMu 70.6 66.1 63.4 65.8 51.9 65.8 AmazInstr 61.2 58.0 57.2 57.4 46.0 57.2 AmazPantry 64.6 59.1 56.5 56.5 44.8 44.8 AmazPantry (20%) AmazPantry 70.2 67.3 62.6 67.2 48.6 48.7 AmazDigiMu 59.5 57.7 54.6 56.2 41.2 57.7 AmazInstr 64.5 63.8 58.0 63.6 40.1 40.3 AmazInstr (20%) AmazInstr 71.5 69.8 62.1 69.7 45.6 48.6 AmazDigiMu 61.3 60.0 53.2 57.8 43.8 60.0 AmazPantry 68.2 64.5 56.3 63.1 44.6 47.6 Table 4: Average F1 macro performance of FRESH models (five runs) with the a priori defined rationale length in the brackets. Bold denotes no significant difference between FRESH and Full-text (t-test; p > 0.05). For clarity, we present F1 scores with standard deviations in Appendix D. is the same rationale length when evaluating on AmazDigiMu using a model trained on AmazInstr. In general, our findings show that in the majority of cases, using HardKuma in out-of-domain data results to comparable performance with their full-text model counterparts. This suggests that HardKuma models can be used in out-of-domain settings, without significant sacrifices in predictive performance whilst also offering faithful rationales . FRESH: Table 4 shows the averaged F1-macro performance across 5 random seeds for FRESH classifiers on inand out-of-domain using TopK rationales. 7 We also include the a priori defined rationale length in parentheses and the predictive performance of the Full-Text model for reference. 8 We first observe that in-domain predictive performance varies across feature attribution approaches with attention-based metrics (    ,  ) outperforming the gradient-based ones ( x  x , IG), largely agreeing with Jain et al. (2020). We also find that    and DeepLift are the feature attribution approaches that lead to the highest predictive performance across all datasets. As we initially hypothesized, performance of FRESH generally degrades when testing on out-of-domain data similarly to the behavior of models 7 For clarity we include standard deviations and Contiguous results in Appendix D 8 When evaluating out-of-domain, we use the average rationale length of the dataset we evaluate on. This makes FRESH experiments comparable with those of HardKuma. trained using the full text. The only exceptions are when using x  x and IG in IMDB. We argue that this is due to these feature attribution methods not being able to identify the appropriate tokens relevant to the task using a rationale length 2% of the original input. Increasing the rationale length to 20% (SST) and 10% (Yelp) also increases the performance. Results also suggest that    and DeepLift outperform the rest of the feature attributions, with    being the best performing one in the majority of cases. In fact when using    or DeepLift, the out-of-domain performance of FRESH is not significantly different to that of models trained on full text (t-test; p-value > 0.05) in 5 cases. For example, a FRESH model trained on AmazPantry and evaluated on AmazInstr records 63.6 F1 macro (using DeepLift) compared to 64.5 obtained by a full-text model. However, this does not apply to the other feature attribution methods (  ; x  x ; IG). To better understand this behavior, we conduct a correlation analysis between the importance rankings using any single feature attribution from (1) a model trained on the same domain with the evaluation data; and (2) a model trained on a different domain (out-of-domain trained model). High correlations suggest that if a feature attribution from an out-of-domain trained model produces similar importance distributions with that of an in-domain model, it will also lead to high predictive performance out-of-domain. Contrary to our initial 6926 assumption we found that the lower the correlation, the higher the predictive performance with FRESH. Results show low correlations when using    and DeepLift (highest FRESH perfor-mance). Surprisingly, IG and x  x (lowest FRESH performance) showed consistently strong correlations across all dataset pairs. Thus, we conclude that lower correlation scores indicate lower attachment to spurious correlations learned during training. We expand our discussion and show results for the correlation analysis in Appendix E. Our findings therefore suggest that using FRESH in out-of-domain settings, can result to comparable performance with a model trained on full-text. However this highly depends on the choice of the feature attribution method . HardKuma vs. FRESH: We observe that HardKuma models are not significantly different compared to models trained on the full text in out-of-domain settings in more cases, when compared to FRESH (9 out of 12 and 5 out of 12 respectively). However, FRESH with    or DeepLift records higher predictive performance compared to HardKuma models (both inand out-of-domain) in all cases . We attribute this to the underlying model architectures, as FRESH uses BERT and HardKuma a biLSTM . As we discussed in 3.2, we attempted using BERT for HardKuma models in the extractor and classifier similar to Jain et al. (2020). However, the performance of HardKuma with BERT is at most comparable to when using a biLSTM similar to findings of Guerreiro and Martins (2021). 5.3 Correlation between Post-hoc Explanation Faithfulness and FRESH Performance We hypothesize that a feature attribution with high scores for sufficiency and comprehensiveness, should extract rationales that result in high FRESH predictive performance. We expect that if our hypothesis is valid, faithfulness scores can serve as early indicators of FRESH performance, both on in-domain and out-of-domain settings. Table 5 shows the Spearman's ranking correlation (  ) between FRESH F1 performance (see Table 4) and comprehensiveness and sufficiency (see Table 2). Correlation is computed using all feature scoring methods for each dataset pair. Results show that only 4 cases achieve statistically significant correlations (p-value < 0.05) with only 3 out-of-domain and mostly between sufficiency and FRESH performance. We do not observe Train Test  FRESH Sufficiency Comprehen. SST SST 0.97 0.15 IMDB 0.36 0.21 Yelp 0.90 0.56 IMDB IMDB 0.69 0.87 SST 0.65 0.23 Yelp 0.92 0.92 Yelp Yelp 0.82 0.55 SST -0.67 -0.67 IMDB 0.87 0.56 AmazDigiMu AmazDigiMu -0.11 0.22 AmazInstr 0.23 0.69 AmazPantry 0.11 0.11 AmazPantry AmazPantry 0.16 0.16 AmazDigiMu 0.05 0.41 AmazInstr 0.16 0.16 AmazInstr AmazInstr 0.79 0.55 AmazDigiMu 0.24 0.67 AmazPantry 0.21 0.20 Table 5: Spearman's ranking correlation (  ) between FRESH performance and comprehensiveness, sufficiency across all feature attribution approaches. Bold denotes statistically significant (p-value  0.05) correlations. high correlations with comprehensiveness which is expected, as comprehensiveness evaluated the rationale's influence towards a model's prediction. Our findings refute our initial hypothesis and suggest that there is no clear correlation across all cases, between post-hoc explanation faithfulness and FRESH predictive performance. Therefore, sufficiency and comprehensiveness scores cannot be used as early indicators of FRESH predictive performance. 6 Qualitative Analysis Table 6 presents examples from a qualitative analysis we performed, aimed at better understanding out-of-domain post-hoc explanations. Rows with highlighted text in blue are from a model trained in the same domain as the presented example (ID), whilst those with highlighted text in red are from models trained on a different domain. Importance scores are computed using scaled attention (    ). In Example (1), we observe that models trained on two closely related tasks (AmazInstr and AmazDigiMu) place more importance to the phrase sound good. For an explanation to be highly comprehensive, the model's prediction after masking the rationale should have a large difference compared to the model's prediction using the full text. On the contrary, the model trained on AmazPantry which has not encountered such phrases during training, mostly focuses on Work great. This is expected as the term sound is not typical of pantry reviews. Similarly, in Example 6927 M Trained On Example (1) AmazInstr (ID) Work great and sound good AmazDigiMu Work great and sound good AmazPantry Work great and sound good (2) AmazPantry (ID) Delicious and at a good price . (2) from the AmazPantry dataset, the in-domain model focuses on a domain-specific word deli-cious. On the contrary, the two models trained on music-related tasks focus on more generic terms such as good and would recommend. In Example (3) the model trained on Yelp focuses mostly on the word behavior, a term we consider more relevant to restaurant reviews rather than movie reviews. In comparison, the other models which are both trained on movie reviews focus both on the term funny. In Example (4), again the two movie-review models focus on more generic terms (i.e. amazing) compared to must taste that the model trained in-domain (i.e. Yelp) identifies as important. Overall, results show that rationales from models applied to a different domain (other than that they were trained for), comprise of terms that are mostly present within the domain they were trained for. This can partly explain the performance of out-of-domain FRESH classifiers. Our assumption, similar to (Adebayo et al., 2020), is that a model's inability to generalize to other domains, is based on the model latching on to specific features from the training dataset. We conducted an extensive empirical study to assess the faithfulness of post-hoc explanations (i.e. using feature attribution approaches) and performance of select-then-predict (i.e. inherently faithful) faithful) models in out-of-domain settings. Our findings highlight, that using sufficiency and comprehensiveness to evaluate post-hoc explanation faithfulness out-of-domain can be misleading. To address this issue, we suggest comparing faithfulness of post-hoc explanations to a random attribution baseline for a more robust evaluation. We also show that select-then-predict models, which are inherently faithful, perform surprisingly well in out-of-domain settings. Despite performance degradation, in many cases their performance is comparable to those of full-text trained models. In future work, we aim to explore methods for improving the evaluation of faithfulness for out-of-domain post-hoc explanations. We would like to thank Katerina Margatina and Tulika Bose for their insightful feedback in a preliminary version of this paper. GC and NA are supported by EPSRC grant EP/V055712/1, part of the European Commission CHIST-ERA programme, call 2019 XAI: Explainable Machine Learning-based Artificial Intelligence.
abstain abstain abstain method result result objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective objective result method objective objective method other other other method other other method other other other other other other other other other method other other other other other other other other other other objective method method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain method method method abstain method abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain result method result abstain objective other other