NLP practitioners often want to take existing trained models and apply them to data from new domains. While fine-tuning or few-shot learning can be used to adapt a base model, there is no single recipe for making these techniques work; moreover, one may not have access to the original model weights if it is deployed as a black box. We study how to improve a black box model's performance on a new domain by leveraging explanations of the model's behavior. Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, then uses a simple calibrator, in the form of a classifier, to predict whether the base model was correct or not. We experiment with our method on two tasks, extractive question answering and natural language inference, covering adaptation from several pairs of domains with limited target-domain data. The experimental results across all the domain pairs show that explanations are useful for calibrating these models, boosting accuracy when predictions do not have to be returned on every example. We further show that the calibration model transfers to some extent between tasks. 1 1 Introduction With recent breakthroughs in pre-training, NLP models are showing increasingly promising performance on real-world tasks, leading to their deployment at scale for translation, sentiment analysis, and question answering. These models are sometimes used as black boxes, especially if they are only available as a service through APIs 2 or if end users do not have the resources to fine-tune the 1 Code available: https://github.com/xiye17/ InterpCalib 2 Google Translate, the Perspective API https:// perspectiveapi.com/ , and MonkeyLearn https:// monkeylearn.com/monkeylearn-api/ being three examples. models themselves. This black-box nature poses a challenge when users try to deploy models on a new domain that diverges from the training domain, usually resulting in performance degradation. We investigate the task of domain adaptation of black box models: given a black box model and a small number of examples from a new domain, how can we improve the model's generalization performance on the new domain? In this setting, note that we are not able to update the model parameters, 6199 which makes transfer and few-shot learning techniques inapplicable. However, we can still make the model more effective in practice by learning a calibrator , or a separate model to make a binary decision of whether the black box model is likely to be correct or not on a given instance. While not fully addressing the domain adaptation problem, calibrating the model can make it more useful in practice, as we can recognize when it is likely to make mistakes (Guo et al., 2017; Kamath et al., 2020; Desai and Durrett, 2020) and modify our deployment strategy accordingly. This paper explores how explanations can help address this task. We leverage black box feature attribution techniques (Ribeiro et al., 2016; Lundberg and Lee, 2017) to identify key input features the model is leveraging, even without access to model internal representations. As shown in Figure 1, we perform calibration by connecting model interpretations with hand-crafted heuristics to extract a set of features describing the reasoning of the model. For the question answering setting depicted in the figure, answers turn out to be more reliable when the tokens of a particular set of tags (e.g., proper nouns) in the question are strongly considered. We extract a set of features describing the attribution values of different tags. Using a small number of examples in the target domain, we can train a simple calibrator for the black box model. Our approach is closely related to the recent line of work on model behavior and explanations. Chan-drasekaran et al. (2018); Hase and Bansal (2020) shows explanations can help users predict model decisions in some ways and Ye et al. (2021) show how these explanations can be semi-automatically connected to model behavior based on manually crafted heuristics. Our approach goes further by using a model to learn these heuristics, instead of handcrafting them or having a human inspect the explanations. We test whether our method can improve model generalization performance on two tasks: extractive question answering (QA) and natural language inference (NLI). We construct generalization settings for 5 pairs of source and target domains across the two tasks. Compared to existing baselines (Ka-math et al., 2020) and our own ablations, we find explanations are indeed helpful for this task, successfully improving calibrator performance among all pairs. We even find settings where explanation-based calibrators outperform fine-tuning the model on target domain data, which assumes glass-box access to the model's parameters. Our analysis further demonstrates generalization of the calibrator models themselves: our calibrator trained on one domain can transfer to another new domain in some cases. Moreover, our calibrator can also substantially improves model performance in the Selective QA setting. Let x = x 1 , x 2 , ..., x n be a set of input tokens and  y = f ( x ) be a prediction from our black box model under consideration. Our task in calibration 3 is to assess whether the model prediction on x matches its ground truth y . We represent this with the variable t , i.e., t  1 { f ( x ) = y } . We explore various calibrator models to perform this task, with our main focus being on calibrator models that leverage explanations in the form of feature attribution . Specifically, an explanation  for the input x assigns an attribution score  i for each input token x i , which represents the importance of that token. Next, we extract features u ( x,  ) depending on the input and explanation, and use the features to learn a calibrator c : u ( x,  )  t for predicting whether a prediction is valid. We compare against baselines that do not use explanations in order to answer the core question posed by our paper's title. Our evaluation focuses on binary calibration, or classifying whether a model's initial prediction is correct. Following recent work in this setting (Ka-math et al., 2020), we particularly focus on domain transfer settings where models make frequent mistakes. A good calibrator can identify instances where the model has likely made a mistake, so we can return a null response to the user instead of an incorrect one. In the remainder of this section, we'll first introduce how we generate the explanations and then how to extract the features u for the input x . (Lund-3 We follow Kamath et al. (2020) in treating calibration as a binary classification task. Devising a good classifier is connected to the goal of accurate estimation of posterior probabilities that calibration has more historically referred to (Guo et al., 2017), but our evaluation focuses on binary accuracy rather than real-valued probabilities. berg and Lee, 2017) for generating explanations for models instead of other techniques that require access to the model details (e.g., integrated gradients (Sundararajan et al., 2017)). The rest of this work only relies on LIME and SHAP to map an input sequence x and a model prediction y to a set of importance weights  . We will briefly summarize the unified framework shared by both methods, and refer readers to the respective papers for additional details. LIME and SHAP generate local explanations by approximating the model's predictions on a set of perturbations around the base data point x . In this setting, a perturbation x  with respect to x is a simplified input where some of the input tokens are absent (replaced with a <mask> token). Let z = z 1 , z 2 , ..., z n be a binary vector with each z i indicating whether x i is present (using value 1) or absent (using value 0), and h x ( z ) be the function that maps z back to the simplified input x  . Both methods seek to learn a local linear classifier g on z which matches the prediction of original model f by minimizing: g ( z ) =  0 + n (cid:88) i =1  i z i  = arg min g (cid:88) z  Z  x ( z )[ f ( h x ( z ))  g ( z )] 2 + ( g ) where  x is a local kernel assigning weight to each perturbation z , and  is the L2 regularizer over the model complexity. The learned feature weight  i for each z i then represents the additive attribution (Lundberg and Lee, 2017) of each individual token x i . LIME and SHAP differ in the choice of the local kernel  x . Please refer to the supplementary materials for details of the kernel. Armed with these explanations, we now wish to connect the explanations to the reasoning we expect from the task : if the model is behaving as we expect, it may be better calibrated. A human might look at the attributions of some important features and decide whether the model is trustworthy in a similar fashion (Doshi-Velez and Kim, 2017). Past work has explored such a technique to compare explanation techniques (Ye et al., 2021), or run studies with human users on this task (Chandrasekaran et al., 2018; Hase and Bansal, 2020). Our method automates this process by learning what properties of explanations are important. We first assign each token x i with one or more human-understandable properties V ( x i ) = { v j } m i j =1 . Each property v j  V is an element in the property space, which includes indicators like POS tags and is used to describe an aspect of x i whose importance might correlate with the model's robustness. We conjoin these properties with aspects of the explanation to render our calibration judgment. Figure 1 shows examples of properties such as whether a token is a proper noun (NNP). We now construct the feature set for the prediction made on x . For every property v  V , we extract a single feature F ( v, x,  ) by aggregating the attributions of the tokens associated with v : F ( v, x,  ) = n (cid:88) i =1 (cid:88)  v  V ( x i ) 1 {  v = v }  i where 1 is the indicator function, and  i is the attribution value. An individual feature represents the total attribution with respect to property v when the model is making the predictions for x . The complete feature set u for x , given as u = { F ( v, x,  ) } v V , summarizes model rationales from the perspective of the properties in V . properties for calibrating QA and NLI models. Segments of the Input (QA and NLI): In both of our tasks, an input sequence can naturally be decomposed into two parts, namely a question and a context (QA) or a premise and a hypothesis (NLI). We assign each token with the corresponding segment name, which yields features like Attributions to Question . POS Tags (QA and NLI): We use tags from the English Penn Treebank (Marcus et al., 1993) to implement a group of properties. We hypothesize that tokens of some specific tags should be more important, like proper nouns in the questions of the QA tasks. If a model fails to consider proper nouns of a QA pair, it is more likely to make incorrect predictions. Overlapping Words (NLI): Word overlap between a premise and a hypothesis strongly affects neural models' predictions (McCoy et al., 2019). We assign each token with the Overlapping property if a token appears in both the premise and the hypothesis, or Non-Overlapping otherwise. Cartesian product of two or more groups. We conjoin Segment and Pos-Tags , which yields higher-level features like Attributions to NNP in Question . Such a feature aggregates attributions of tokens that are tagged with NNP and also required to be in the question (marked with orange). We train the calibrator on a small number of samples in our target domain. Each sample is labeled using the prediction of the original model compared to the ground truth. Using our feature set F ( v, x,  ) , we learn a random forest classifier, shown to be effective for a similar data-limited setting in Kamath et al. (2020), to predict t (whether the prediction is correct). This classifier returns a score, which overrides the model's original confidence score with respect to that prediction. In Section 4, we discuss several baselines for our approach. As we vary the features used by the model, all the other details of the classifier and setup remain the same. Our task setup involves transferring from a source domain/task A to a target domain/task B. Figure 2 shows the data conditions we operate in. Our primary experiments focus on using our features to either calibrate or selectively answer in the black box setting (right side in Figure 2). In this setting, we have a black box model trained on a source domain A and a small amount of data from the target domain B. Our task is to train a calibrator using data from domain B to identify instances where the model potentially fails in the large unseen test data in domain B. We contrast this black box setting with glass box settings (left side in Figure 2), where we directly have access to the model parameters and can fine-tune on domain B or train on B from scratch. English Question Answering We experiment with domain transfer from SQUAD (Rajpurkar et al., 2016) to three different settings: SQUAD-ADV (Jia and Liang, 2017), HOTPOTQA (Yang et al., 2018), and TRIVIAQA (Joshi et al., 2017). SQUAD-ADV is an adversarial setting based on SQUAD , which constructs adversarial QA examples based on SQUAD by appending a distractor sentence at the end of each example's context. The added sentence contains a spurious answer and usually has high surface overlapping with the question c B l ack -B o x C a li b r a t i o n Base Model Calibration Test S e l ec t i v e QAA A C A B ? so as to fool the model. We use the ADDSENT setting from Jia and Liang (2017). Similar to SQUAD , HOTPOTQA also contains passages extracted from Wikipedia, but HOTPOTQA asks questions requiring multiple reasoning steps, although not all questions do (Chen and Durrett, 2019). TRIVIAQA is collected from Web snippets, which present a different distribution of questions and passages than SQUAD . For HOTPOTQA and TRIVIAQA, we directly use the pre-processed version of dataset from the MRQA Shared Task (Fisch et al., 2019). English NLI For the task of NLI, we transfer a model trained on MNLI (Williams et al., 2018) to MRPC (Dolan and Brockett, 2005) and QNLI (Wang et al., 2019), similar to the settings in Ma et al. (2019). QNLI contains a question and context sentence pair from SQUAD , and the task is to verify whether a sentence contains the answer to the paired question. MRPC is a paraphrase detection dataset presenting a binary classification task to decide whether two sentences are paraphrases of one another. Note that generalization from MNLI to QNLI or MRPC not only introduces shift in terms of the distribution of the input text, but in terms of the nature of the task itself, since QNLI and MRPC aren't strictly NLI tasks despite sharing some similarity. Both are binary classification tasks rather than three-way. Baselines We compare our calibrator against existing baselines as well as our own ablations. KAMATH (Kamath et al., 2020) (for QA only) is a baseline initially proposed to distinguish out-of-distribution data points from in-domain data points in the selective QA setting (see Section 5), but it can also be applied in our settings. It trains a random forest classifier to learn whether a model's prediction is correct based on several heuristic features, including the probabilities of the top 5 predictions, the length of the context, and the length of the predicted answer. Since we are calibrating black box models, we do not use dropout-based features in Kamath et al. (2020). CLSPROBCAL (for NLI only) uses more detailed information than MAXPROB : it uses the predicted probability for Entailment , Contradiction , and Neutral as the features for training a calibrator instead of only using the maximum probability. BOWPROP adds a set of heuristic property features on top of the KAMATH method. These are the same as the features used by the full model excluding the explanations . We use this baseline to give a baseline for using general shape features on the inputs not paired with explanations. Implementation of Our Method We refer our explanation-based calibration method using explanations produced by LIME and SHAP as LIMECAL and SHAPCAL respectively. We note that these methods also take advantages of the bag-of-word features in BOWPROP . For QA, the property space is the union of low-level Segment and Segment  Pos-Tags . For NLI, we use the union of Segment and Segment  Pos-Tags  Overlapping Words to label the tokens. Setup We train a ROBERTA (Liu et al., 2019) QA model on SQUAD as the base model, which achieves 85.5 exact match and 92.2 F1 score. For the experiments on HOTPOTQA and TRIVIAQA, we split the dev set, sample 500 examples for training, and leave the rest for testing. 4 For experiments on SQUAD-ADV , we remove the unmodified data points in the ADD-SENT setting and also use 500 examples for training. For the experiments across all pairs, we randomly generate the splits, test the methods 20 times, and average the results to alleviate the influence of randomness. Metrics In addition to calibration accuracy ( ACC ) that measures the accuracy of the calibrator, we also use the area under coverage-F1 curve ( AUC ) to evaluate the calibration performance for QA tasks in particular. The coverage-F1 curve (Figure 3) plots the average F1 score of the model achieved when the model only chooses to answer varying fractions (coverage) of the examples ranked by the calibrator-produced confidence. A better calibrator should assign higher scores to the questions that the models are sure of, thus resulting in higher area under the curve; note that AUC of 100 is impossible since the F1 is always bounded by the base model when every question is answered. We additionally report the average scores when answering the top 25%, 50%, and 75% questions, for a more intuitive comparison of the performance. Results Table 1 summarizes the results for QA. First, we show that explanations are helpful for calibrating black box QA models out-of-domain. Our method using LIME substantially improves the calibration AUC compared to KAMATH by 7.1, 2.1 and 1.4 on SQUAD-ADV , TRIVIAQA, and HOTPOTQA, respectively. In particular, LIMECAL achieves an average F1 score of 92.3 at a coverage of 25% on SQUAD-ADV , close to the performance of the base model on original SQUAD examples. Our explanation-based approach is effective at identifying the examples that are robust with respect to the adversarial attacks. PROP performs on par with or only slightly better than KAMATH . These results show that connecting explanations with annotations is a path towards building better calibrators. Finally, we compare the performance of our methods based on different explanation techniques. LIMECAL slightly outperforms SHAPCAL in all three settings. As discussed in Section 2.1, SHAP assigns high instance weights to those perturbations with few activated features. While such a choice of the kernel is effective in tasks involving tabular data (Lundberg and Lee, 2017), this might not be appropriate for the task of QA when such perturbations may not yield meaningful examples. Setup Our base NLI model is a ROBERTA classification model trained on MNLI, which achieves 87.7% accuracy on the development set. We collapse contradiction and neutral into non-entailment when evaluating on QNLI and MRPC. We continue using random forests as the calibrator model. We evaluate the generalization performance on the development sets of QNLI and MRPC. Similar to the settings in QA, we use 500 examples to train the calibrator and test on the rest for each of the 20 random trials. Metrics Because QNLI and MRPC are binary classification tasks, predicting whether a model is correct (our calibration setting) is equivalent to the original prediction task. We can therefore measure calibrator performance with standard classification accuracy and AUC. Results We show results on NLI tasks in Table 2. The base MNLI model utterly fails when transferring to QNLI and MRPC and achieves an accuracy of 49% and 57%, respectively, whereas the majority class is 50% (QNLI) and 65% (MRPC). With heuristic annotations, BOWPROP is able to solve 74% of the QNLI instances and 72% of the MRPC instances. Our heuristic itself is strong for QNLI compared to MAXPROB . LIMECAL is still the best in both settings, moderately improving accuracy by 1% and 2% over BOWPROP using explanations. The results on NLI tasks suggest our method can still learn useful signals for indicating model reliability even if the underlying tasks are very different. transfer settings. Is the knowledge of a calibrator learned on some initial domain transfer setting, e.g., SQuAD  TRIVIAQA, generalizable to another transfer setting, e.g.  HOTPOTQA? This would enable us to take our basic QA model and a calibrator and apply that pair of models in a new domain without doing any new training or adaptation. We explore this hypothesis on QA. 5 For comparison, we also give the performance of a ROBERTA -model first finetuned on SQUAD and then finetuned on domain A (ADAPT , Figure 2). ADAPT requires access to the model architecture and is an unfair comparison for other approaches. We show the results in Table 3. None of the approaches generalize between SQUAD-ADV and the other domains (either trained or tested on SQUADADV ), which is unsurprising given the synthetic and very specific nature of SQUAD-ADV . Between TRIVIAQA and HOTPOTQA, both the LIMECAL and KAMATH calibrators trained on one 5 We also tested the hypothesis on the NLI-paraphrase transfer, but did not see evidence of transferability there, possibly due to the fact that these tasks fundamentally differ. domain can generalize to the other, even though BOWPROP is not effective. Furthermore, our LIMECAL exhibits a stronger capability of generalization compared to KAMATH . We then compare LIMECAL against ADAPT . ADAPT does not always work well, which has also been discussed in Kamath et al. (2020); Talmor and Berant (2019). ADAPT leads to a huge drop in terms of performance when being trained on HOTPOTQA and tested on TRIVIAQA, whereas LIMECAL is the best in this setting. From TRIVIAQA to HOTPOTQA, ADAPT works well, but LIME is almost as effective. ity across the two realistic QA tasks. We believe this can be attributed to the features used in the explanation-based calibrator. Although the task is different, the calibrator can rely on some common rules to decide the reliability of a prediction. Impacts of Training Data Size Calibrating a model for a new domain becomes cumbersome if large amounts of annotated data are necessary. We experiment with varying the amount of training data the calibrator is exposed to, with results shown in Table 4. Our explanation-based calibrator is still the best in every setting with as few as 100 examples. With 100 examples, KAMATH and BOWPROP perform worse than the MAXPROB baseline on TRIVIAQA and HOTPOTQA, indicating that more data is needed to learn to use their features. Throughout this work, we have assumed a black box model that cannot be fine-tuned on a new domain. In this section, we compare calibration-based approaches with glass-box methods that require access to the model architectures and parameters. We evaluate two glass-box methods in two different settings (Figure 2): (1) finetuning a base ROBERTA model (FINETUNEROBERTA ), which needs access to the model's architecture but not parameters; and (2) finetuning a base QA/NLI model, which requires both model architectures as well as parameters. All these models are finetuned with 500 examples, the same as LIMECAL . We also give the performance of a model trained with full in-domain training data for different tasks as references (INDOMAIN QA/NLI). We present the model performance (measured with Exact Match and F1 for QA and Acc for NLI) and calibration results in Table 5. Note that there are no calibrators for glass box methods, so we only report AUC scores for calibration performance. On QA tasks, the limited training data is not sufficient for successfully finetuning a ROBERTA model. Consequently, FINETUNEROBERTA does not achieve credible performance. Finetuning a base QA model greatly improves the performance, surpassing LIMECAL on SQUAD-ADV and HOTPOTQA. However, we still find that on TRIVIAQA, LIMECAL slightly outperforms ADAPT . This is a surprising result, and shows that explanation-based calibrators can still be beneficial in some scenarios, even if we have full access to the model. On NLI tasks that are substantially easier than QA, finetuning either a ROBERTALM model or a base NLI model can reach an accuracy of roughly 80%. Our explanation-based approach largely lags glass-box methods, likely because the base NLI model utterly fails on QNLI (50.5% accuracy) and MRPC (55.0% accuracy) and does not grant much support for the two tasks. Nonetheless, the results on NLI still support our main hypothesis: explanations can be useful for calibration. Our results so far have shown that a calibrator can use explanations to help make binary judgments of correctness for a model running in a new domain. We now test our model on the selective QA setting from Kamath et al. (2020) (Figure 2). This experiment allows us to more directly compare with prior work and see performance in a setting where in-domain (ID) and out-of-domain (OOD) examples are mixed together. Given a QA model trained on source domain data, the goal of selective QA is to train a calibrator on a mixture of ID source data and known OOD data, and test the calibrator to work well on a 6206 Known \ Unknown SQ-ADVTRIVIAHOTPOTSQADVMAXPROB 85.0 88.7 87.5 KAMATH 88.8 89.5 88.9 BOWPROP 91.5 90.6 89.0 LIMECAL 94.5 91.7 91.9 TRIVIAMAXPROB 85.0 88.7 87.6 KAMATH 85.6 91.9 88.7 BOWPROP 85.3 92.1 89.9 LIMECAL 90.9 92.5 92.1 HOTPOTMAXPROB 85.0 88.7 87.6 KAMATH 86.1 91.4 89.4 BOWPROP 85.1 91.8 91.6 LIMECAL 91.7 92.3 92.5 Table 6: Area under Coverage-F1 curve in the Selective QA setting. We follow the similar experimental setup as in Kamath et al. (2020). The detailed setting is included in the supplementary material. Results As shown in Table 6, similar to the main QA results. Our explanation-based approach, LIMECAL , is consistently the best among all settings. We point out our approach outperforms KAMATH especially in settings that involve SQUADADV as known or unknown OOD distribution. This can be attributed the similarity between SQUAD and SQUAD-ADV which can not be well distinguished with features used in KAMATH ( Context Length, Answer Length , and etc.). The strong performance of our explanation-based approach in the selective QA setting further verifies our assumption: explanation can be useful and effective for calibrating black box models. Our approach is inspired by recent work on the simulation test (Doshi-Velez and Kim, 2017), i.e., whether humans can simulate a model's prediction on an input example based on the explanations. Simulation tests have been carried out in various tasks (Ribeiro et al., 2018; Nguyen, 2018; Chan-drasekaran et al., 2018; Hase and Bansal, 2020) and give positive results in some tasks (Hase and Bansal, 2020). Our approach tries to mimic the process that humans would use to judge a model's prediction by combining heuristics with attributions instead of having humans actually do the task. like machine translation (Bojar et al., 2017), question answering (Kamath et al., 2020; Zhang et al., 2021), constituency parsing (Charniak and Johnson, 2005; Fossum and Knight, 2009) and semantic parsing (Yin and Neubig, 2019). The work of Rajani and Mooney (2018) in VQA is most relevant to ours; they also use heuristic features, but we further conjoin heuristic with model attributions. Our meta-feature set is derived from the presence of certain properties, which is similar to the con-cepts used in concept-based explanations (Ghor-bani et al., 2019; Mu and Andreas, 2020), but we focus on using them for estimating model performance rather than explaining a prediction. Our work addresses the problem of calibration (Guo et al., 2017; Desai and Durrett, 2020), which is frequently framed in terms of models' output probabilities. Past work has attempted to tackle this problem using temperature scaling (Guo et al., 2017) or label smoothing (Pereyra et al., 2017), which adjust confidence scores for all predictions. In contrast, we approach this issue by applying a classifier leveraging instance-specific explanations. Past work on generalizing to out-of-domain distribution in NLP largely focuses on using unlabeled data from the target domain and requires finetuning a model (Ma et al., 2019; Ramponi and Plank, 2020; Guo et al., 2020), whereas we improve OOD performance of strictly black-box models. Limitations Despite showing promising results in improving model generalization performance, our attribution-based approach does suffer from intensive computation cost. Using either LIME or SHAP to generate attributions requires running inference a fair number of perturbations when the input size is large (see Appendix for details), which limits our method's applicability. But this doesn't undermine the main contribution of this paper, answering the question in the title, and our approach is still applicable as-is in the scenarios where we pay for access to the model but not per query. Conclusion We have explored whether model attributions can be useful for calibrating black box models. The answer is yes . By connecting attributions with human heuristics, we improve model generalization performance on new domains and tasks. Besides, it exhibits promising generalization performance in some settings (cross-domain generalization and Selective QA). Thanks to the anonymous reviewers for their helpful feedback. This work was partially supported by NSF Grant IIS-1814522, a gift from Salesforce Inc, and a gift from Amazon.
abstain abstain objective objective method abstain result abstain other abstain abstain objective method abstain method objective method result abstain method method abstain abstain method result method result result objective result abstain abstain abstain abstain abstain method method method method method abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method other method abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain other method abstain abstain abstain abstain abstain abstain method other method other method method abstain other objective abstain result result objective objective abstain objective abstain other other