Modern deep learning models are notoriously opaque, which has motivated the development of methods for interpreting how deep models predict. This goal is usually approached with attribution method, which assesses the influence of features on model predictions. As an explanation method, the evaluation criteria of attribution methods is how accurately it reflects the actual reasoning process of the model (faithful-ness). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to demonstrate their arguments. However, some crucial logic traps in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison. This paper systematically reviews existing methods for evaluating attribution scores and summarizes the logic traps in these methods. We further conduct experiments to demonstrate the existence of each logic trap. Through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps. The opaqueness of deep models has grown in tandem with their power (Doshi-Velez and Kim, 2017), which has motivated efforts to interpret how these black-box models work (Sundararajan et al., 2017; Belinkov and Glass, 2019). Post-hoc explanation aims to explain a trained model and reveal how the model arrives at a decision (Jacovi and Goldberg, 2020; Molnar, 2020). This goal is usually approached with attribution method, which assesses the influence of features on model predictions as shown in Figure 1. Recent years have witnessed an increasing number of attribution methods being developed. For example, Erasure-based method calculate attribution scores by measuring the change of output after removing target features (Li et al., 2016; Feng et al., 2018; Chen et al., 2020); Gradient-based method uses gradients to study the influence of features on model predictions (Sun-dararajan et al., 2017; Wallace et al., 2019; Hao et al., 2020); Meanwhile, these methods also received much scrutiny, arguing that the generated attribution scores are fragile or unreliable (Alvarez-Melis and Jaakkola, 2018; Pruthi et al., 2020; Wang et al., 2020; Slack et al., 2020). As an explanation method, the evaluation criteria of attribution methods should be how accurately it reflects the true reasoning process of the model (faithfulness), not how convincing it is to humans (plausibility) Jacovi and Goldberg (2020). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to support their arguments, some of which appear valid and are widely used in the research field. For example, meaningful perturbation is used for making comparison in many works (Samek et al., 2016; Chen et al., 2019; DeYoung et al., 2020; Chen et al., 2020; Kim et al., 2020). The philosophy of meaningful perturbation is simple, i.e., modifications to the input instances, which are in accordance with the generated attribution scores, can bring about significant differences to model predictions if the attribution scores are faithful to the target system. works, causing inaccurate evaluation and unfair comparison. For example, we found that we can manipulate the evaluation results when using meaningful perturbation to make comparisons: by choosing the modification strategy, we can assign any of the three candidate attribution methods as the best method. The neglect of these traps has damaged the community in many aspects: First, the existence of logic traps will lead to an inaccurate evaluation and unfair comparison, making the conclusions unreliable; Second, the wide use of evaluation metrics with logic traps brings pressure to newly proposed works, requiring them to compare with other works using the same metrics; Last, the over-belief in existing evaluation metrics encourages efforts to propose more accurate attribution methods, notwithstanding the evaluation system is unreliable. In this paper, we systematically review existing methods for evaluating attribution scores and categorize them into classes. We summarize the logic traps in these methods and further conduct experiments to demonstrate the existence of each logical trap. Though strictly accurate evaluation metrics for attribution methods might be a unicorn which will likely never be found, we should not just ignore logic traps in existing evaluation methods and draw conclusions recklessly. Through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps. Evaluation 1 verifies the validity of the attribution scores by comparing them with the human problem-solving process. In this evaluation, works (e.g., Murdoch et al. (2018); Kim et al. (2020); Sun-dararajan et al. (2017)) often give examples consistent with human understandings to demonstrate the superiority of their proposed method. For example, as shown in Table 1, Murdoch et al. (2018) shows heat maps for a yelp review generated by different attribution techniques. They argue that the proposed method: Contextual decomposition, Method Heat Map Leave One Out used to be my favorite Integrated gradients used to be my favorite Contextual decomposition used to be my favorite Legend: Very Negative Negative Neutral Positive Very Positive Table 1: Heat maps for a portion of a yelp review generated by different attribution techniques. is better than others because only it can identify favorite' as positive and used to be' as negative, which is consistent with human understandings. Furthermore, resorting to human-annotated explanations, works can also evaluate attribution methods quantitatively in evaluation 1. For example, the SST-2 (Socher et al., 2013) corpus provides not only sentence-level labels, but also five-class word-level sentiment tags ranging from very negative to very positive. Thus, many works (Lei et al., 2016; Li et al., 2016; Tsang et al., 2020; Kim et al., 2020) perform quantitative evaluation of attribution scores by comparing them with the word-level tags in SST-2. First, we cannot completely deny the rationality of evaluation 1. Since many attribution methods work without any human-annotated information, such as erasure-based and gradient-based methods, the similarity between human-annotated explanations and generated attribution scores can be seen as drawing from the reasoning process of target models. However, since the deep model often rely on unreasonable correlations, even when producing correct predictions, attribution scores preposterous to humans may reflect the reasoning process of the deep model faithfully. Thus we cannot deny the validity of an attribution score through its inconsistency to human-annotated explanations and cannot use human-annotated explanations to conduct a quantitative evaluation. In experiment 1, we give an example to demonstrate that the model might rely on correlations inconsistent with human understandings to get the prediction: though trained with questions, a question-answering model could maintain the same prediction for a large ratio of samples when the We experiment on RACE (Lai et al., 2017), a large-scale question-answering dataset. As shown in Table 2, RACE requires the model to choose the right answer from candidate options according to the given question and document. Document: ...Many people optimistically thought industry awards for better equipment would stimulate the production of quieter appliances. It was even suggested that noise from building sites could be alleviated ... Question: What was the author's attitude towards the industry awards for quieter? Options: A. suspicious B. positive C. enthusiastic D. indifferent Table 2: An sample taken from RACE dataset. We first train a model with BERT base (De-vlin et al., 2019) as encoder 1 with questions, and achieve 65.7% accuracy on the development set. Then, we replace the development set questions with empty strings and feed them into the trained model. Surprisingly, the trained MRC model maintained the original prediction on 64.0% of the test set samples (68.4% on correctly answered samples and 55.4% on wrongly answered samples). Moreover, we analyze the model confidence change in these unchanged samples, where the probability on the predicted label is used as the confidence score. As shown in Figure 2, most of the samples have confidence decrease smaller than 0.1, demonstrating question information are not essential for the model to get predictions in these samples. Since question information is usually crucial for humans to answer the question, attribution scores faithfully reflect the reasoning process of this model may be inconsistent with human annotations. Thus, it is improper to use human-annotation 1 Our implementations of experiment 1 and experiment 2 are based on the Huggingface's transformer model hub ( https://github.com/huggingface/ transformers ), and we use its default model architectures without change for corresponding tasks. explanations as the ground truth to evaluate attribution methods. Most existing methods for quantitatively evaluating attribution scores can be summarized as evaluations based on meaningful perturbation . The philosophy of meaningful perturbation is simple, i.e., modifications to the input instances, which are in accordance with the generated attribution scores, can bring about significant differences to the target model's predictions if the attribution scores are faithful to the target model. For example, Samek et al. (2016); Nguyen (2018); Chen and Ji (2020) use the area over the perturbation curve (AOPC) (Samek et al., 2016) as evaluation metrics. Specifically, given the attribution scores of a set of features, AOPC(k) modifies the top k% features and calculates the average change in the prediction probability as follows, AOP C ( K ) = 1 NN (cid:88) i =1 (cid:110) p ( y | x i )  p ( y |  x ( k ) i ) (cid:111) where  y is the predicted label, N is the number of examples, p ( y | ) is the probability on the predicted class, and  x ( k ) i is modified sample. Higher AOPCs is better, which means that the features chosen by attribution scores are more important; Feng et al. (2018); Petsiuk et al. (2018); Kim et al. (2020) use area under the curve (AUC) to evaluate attribution scores. As shown in Figure 3, AUC plots a prediction probability curve about modified feature numbers where features are modified in order of attribution scores. The argument is if attribution scores are faithful, then the curve will drop rapidly, resulting in a small area under a curve. Besides these works, a lot of works (Shrikumar et al., 2017; Chen et al., 2019; Nguyen, 2018; DeYoung et al., 2020; Chen et al., 2020; Hao et al., 2020; Jiang et al., 2021) use similar metrics to perform evaluation and comparisons. The main difference between evaluation metrics in these works is the difference in the modification strategy. For example, to evaluate word-level attribution scores for SST-2, Chen et al. (2020) uses deleting tokens as modification while Kim et al. (2020) uses replacing tokens with tokens sampled from the distribution inferred by BERT. Evaluation methods based on meaningful perturbation can be seen as an attribution method too. For example, AOPC(k), which assesses the importance of k% features, can be seen as an attribution method calculating an attribution score for k% features. Specifically, when using deleting tokens as modification and narrowing the k% to one token, AOPC(k) degenerates into the basic attribution method: leave-one-out (Li et al., 2016). Thus, evaluation 2 uses an attribution method as the ground truth to evaluate the target attribution method, which measures the similarity between two attribution methods instead of faithfulness. Since meaningful perturbation assesses the importance of features by calculating output change after modifications, its results are mainly depend on how to conduct the modifications, which means different modification strategies might lead to different evaluation results. Evaluation 2 is widely used to compare attribution methods in the research field. Accordingly, the neglect of logic trap 2 has led to a high risk of unfair comparisons and unreliable conclusions. In experiment 2, we give an example of unfair comparisons in evaluation 2: the more similar the target attribution method to the modification strategy, the better the evaluation results. Specifically, by modifying the modification strategies in APOC and AUC, we can assign any of the three candidate attribution methods as the best method. We conduct experiments on on widely used SST-2 task of the GLUE benchmark (Wang et al., 2019)), and use BERT base as encoder to build the target model 1 Figure 4: The overview of LOO, Marg and HEDGE. Attribution Methods We experiment with three attribution methods: leave-one-out (LOO) (Li et al., 2016), HEDGE (Chen et al., 2020) and Marg (Kim et al., 2020). The schemes of these attribution methods are shown in Figure 4, LOO assign attribution scores to the target word good' by deleting it from the sentence and observing change in the model predictions; Marg marginalizes the target word good' out considering the likelihoods of all candidate words, which uses BERT to measure the likelihoods of candidate words to replace the target word; HEDGE builds hierarchical explanations by recursively detecting the weakest interactions and then dividing large text spans into smaller ones. HEDGE assign attribution scores to spans by using '[PAD]' token to replace other words in a sentence and measuring how far the prediction is to the prediction boundary. Evaluation metrics and Results We first evaluate three attribution methods with metrics drawn from Marg and HEDGE papers. Marg uses AUC as evaluation metrics and modifies words by gradually replacing them with a token sampled from the distribution inferred by BERT, denoted as AUC rep ; HEDGE uses AOPC as evaluation metrics and modifies words by deleting them directly, denoted as AOPC del . Both papers modify 20% of words in the sentence. The results are shown in Table 3. As shown in Table 3, Marg performs best in AUC rep while LOO performs best in AOPC del . Since the 5914 Method/Metric AOPC del  AUC rep  AOPC rep  AUC del  AOPC pad  AUC pad  LOO 0.541 0.666 0.378 0.526 0.935 0.896 HEDGE 0.466 0.702 0.324 0.638 0.978 0.984 Marg 0.477 0.617 0.391 0.588 0.928 0.874 Table 3: Evaluation results of three attribution methods. modification strategy of AOPC del is consistent with LOO, and that of AUC rep is most similar to Marg, the evaluation results are consistent with the inference in logic trap 2: the more similar the target evaluated method to the modification strategy, the better the evaluation results. Manipulate Evaluation Results We further conduct ablation experiments by changing the modification strategies in AOPC del and AUC rep . Concretely, we switched perturbing strategy in AOPC del and AUC rep and get new evaluation metrics: AOPC rep and AUC del . As shown in Table 3, different from the initial results, Marg performs best in APOC metric while LOO performs best in AUC metric. The opposite results demonstrate that evaluation results mainly depend on the modification strategies, and we can manipulate evaluation results by changing them. Moreover, we note that HEDGE performs worst in all four evaluation metrics. Thus, we further customize the modification strategy to HEDGE's advantage: padding unimportant features according to the attribution scores, denoted as AOPC pad and AUC pad . Not surprisingly, results in Table 3 show that HEDGE perform best in customized metrics. Summarization Because of the existence of logic trap 2, we can manipulate the evaluation results in evaluation 2 by changing the modification strategies, assigning any of the three candidate attribution methods as the best method. In fact, because we cannot simply assign a modification strategy as faithful, we should not use evaluation 2 to quantitatively evaluate attribution scores and make comparisons. Since the wide use of evaluation 2, the neglect of logic trap 2 has negatively impacted the research field for a long time. First, it brings a risk of unfair comparisons: works can customize evaluation metrics to their advantage and thus achieve the best performance. Second, the wide use of evaluation 2 also brings pressure to new proposed works, forcing them to make comparisons to others in such evaluation. In this evaluation, works evaluate attribution methods by examining the consistency of attribution scores for similar inputs. The philosophy of Evaluation 3 is that semantically similar inputs which share the same model predictions should have similar attribution scores if the attribution method is reliable. Evaluation 3 is often used to disprove the effectiveness of attribution methods by searching for counterexamples. For example, ExplainFooler (Sinha et al., 2021) attacks Integrated Gradients and (Sundararajan et al., 2017) and LIME (Sundararajan et al., 2017), which are two popular attribution methods in NLP, by searching adversarial sentences with different attribution scores. As shown in Figure 5, these adversarial sentences are semantically similar to the original sentence and share the same model predictions. However, the attribution scores of these sentences are very different from that of the original sentence. Sinha et al. (2021) observes the rank order correlation drops by over 20% when less than 10% of words are changed on average and thus draws the conclusion that Integrated Gradients and LIME are fragile. A lot of works (Alvarez-Melis and Jaakkola, 2018; Kindermans et al., 2019; Ghorbani et al., 2019; Ding and Koehn, 2021; Sinha et al., 2021) use evaluation 3 to examine the validity of existing attribution methods. For example, Ghorbani et al. (2019) argues that interpretations of neural networks are fragile by showing that systematic perturbations can lead to different interpretations without changing the label; Alvarez-Melis and Jaakkola (2018) argues that a crucial property that attribution methods should satisfy is robustness to local perturbations of the input. Logic Trap 3: The change in attribution scores maybe because the model reasoning process is really changed rather than the attribution method is unreliable. When solving similar samples like those shown in Figure 5, humans tend to use similar reasoning methods. However, deep models are not as robust enough as humans and often rely on unreasonable correlations. Semantically similar texts often cause different reasoning processes in deep models. For example, it is well known that deep models are vulnerable to adversarial samples (Goodfellow et al., 2015; Papernot et al., 2016). By deliberately adding some subtle interference that people cannot detect to the input sample, the target model will give a different prediction with high confidence. The success in adversarial attacks on deep models demonstrates similar inputs for humans can share very different reasoning processes in deep models. The main difference between attribution-attacking methods and model-attacking is that attribution-attacking methods require the model to give the same prediction for adversarial samples. However, giving the same prediction is very weak to constraint model reasoning because deep models have compressed the complicated calculation process into limited classes in the prediction. For example, there is always half probability of giving the same prediction for a binary classification task even with totally random reasoning. Thus, it is no surprise that attribution-attacking methods can find adversarial samples which share the same prediction label to the original sample yet have different attribution scores. The logic trap in evaluation 3 is that the change in attribution scores may be because the model reasoning process is really changed rather than the attribution method is unreliable. As shown in Fig-Figure 6: We use lines connecting inputs and outputs to represent the model reasoning process. (a) is a successful attack on the target model while (b) might be regarded as a successful attack on attribution methods, falling into the logic trap 3. ure 6. (b), an attribution method should generate different attribution scores for the original and adversarial samples if it faithfully reflects the model reasoning. However, it will be regarded as fragile or unreliable in evaluation 3. Unfortunately, existing works ignore this logic trap and propose various methods to attack attribution methods. Since the high susceptibility of deep models to adversarial samples, not surprisingly, all of these works get the same conclusion: existing attribution methods are fragile or unreliable. In experiment 3, we demonstrate that deep models can assign the same label to semantically similar samples yet use different reasoning. We experiment on widely used SST-2 and MNLI tasks of the GLUE benchmark (Wang et al., 2019)). MNLI requires the model to predict whether the premise entails the hypothesis, contradicts it, or is neutral. Model Since the attribution methods are defaulted as unreliable in evaluation 3, we cannot use existing attribution methods to judge whether the model reasoning is different. To solve the problem, we use a two-stage model framework, where the model first extracts a subset of inputs and gives prediction based only on the subset information. This way, we can observe whether the model reasoning is changed from the chosen subset, i.e., different subsets means the model chooses to use different 5916 Figure 7: The overview of the model scheme, which consists of two components: extractor and classifier. The overview of our model is shown in Figure 7. To guarantee that only the subset information is included in the classifier, we discretely select the words and pass words instead of the hidden states of the extractor to the classifier. Since gradients do not flow through discrete samples, we resort to HardKuma (Bastings et al., 2019) to jointly train the model, which gives support to binary outcomes. HardKuma allows for setting the percentage of selected words and is proved more effective and stable than REINFORCE (Williams, 1992) in such scenarios. We set the selection ratio as 20% for SST-2 and 40% for MNLI because larger ratios will not cause further performance improvement. Finally, We get 85.6% accuracy on SST-2 and 66.2/65.5 % accuracy on MNLI-m/mm. Adversarial Attack Method We use TextFooler (Jin et al., 2020) to generate adversarial samples. We use the same settings to Jin et al. (2020) to guarantee the semantical similarity of adversarial samples. The only difference is that we search for samples with minimal similarity in the selected subset instead of the model prediction. We guarantee that the model makes the same predictions, which is often used as the constraint for model reasoning in evaluation 3. We generate adversarial samples with 10% and 20% perturbation ratios. Results We use F1-score to compute the similarity score between subsets and report the Macro-averaging F1-score of the whole development set. 2 Note that similar subsets are regarded as a necessary condition rather than a sufficient condition for the similar model reasoning process. A lower score is better, reflecting a larger difference in selected subsets. Note that since some words in original samples are replaced with their synonyms in adversarial samples, synonyms are seen as identical to their original words when evaluating. We evaluate all samples in the SST-2 development set and the first 1000 samples in MNLI-m/mm development sets. The results are shown in Table 4 As shown in Table 4, though semantically similar to the original samples and share the same model predictions, the adversarial samples can have subsets with low similarity to the original subset. Moreover, with a 10% perturbation ratio, 31.8% of samples in SST-2 have an adversarial subset with none word overlap with the original subset. This result increases to 50.5% with a 20% perturbation ratio. With no overlap between the two subsets, there is no way we can hypothesis the adversarial samples share similar model reasoning to the original samples. Summarization Though evaluation 3 seems reasonable, sharing similar semantics and the same model predictions is a weak constraint for similar model reasoning. Thus the change in attribution scores may come from different model reasoning instead of the instability of attribution methods. Because of deep models' high sensitivity to adversarial samples, works resorting to evaluation 3 all get the same conclusion that existing attribution methods are fragile or unreliable. We argue we should find a more strict constraint for model reasoning first instead of ignoring logic trap 3 and disproving attribution methods recklessly. Besides resorting to methods in evaluation 3, there are works (Jain and Wallace, 2019; Wang et al., 2020; Slack et al., 2020) disprove the reliability of attribution methods by replacing the target model which attribution methods should work on. For example, Slack et al. (2020) trains an adversarial classifier e ( x ) to distinguish whether the inputs have been perturbed or not and then uses a different sub-model to process perturbed instances. Specifically, if we want to attack the LOO method, we can build a loo set from the original dataset and train e ( x ) in the following form: e ( x ) = (cid:40) f ( x ) , if x  original set  ( x ) , if x  loo set This way,  ( x ) , a model irrelevant to model predictions, is used when using LOO to calculate attribution scores, making generated attribution scores meaningless. Slack et al. (2020) assert that results of perturbation-based attribution methods such as LIME and SHAP (Lundberg and Lee, 2017) are easily attacked by their method. Similarly, Wang et al. (2020) add an extra model to the original model, which has uniform outputs but large gradients for some particular tokens such as CLS' in BERT. Since the extra model generates uniform outputs, it will not affect predictions of the original model. However, the extra model's gradients will add to the original model and thus can confuse gradient-based attribution methods. The attack methods in Section 3.1 fool the attribution methods through designing a special structure and require attribution methods to be used in a black-box way. In this setting, the attribution methods are easily attacked and generate meaningless results. However, the question is: as a tool to help humans understand how deep models work, is it necessary to use attribution methods in a black-box way? Take the linear model as an example. The linear model is regarded as a white-box model, and humans don't need attribution methods to understand how it works. However, the understanding of a linear model is based on the analysis of its calculation process. Meanwhile, the deep model is regarded as a black-box model because its calculation process is too complicated to understand for humans, not because its calculation process is inaccessible. Thus, we believe there are no compelling reasons to require attribution methods to be used in a black-box way. The attacks in Wang et al. (2020); Slack et al. (2020) will fail when humans use attribution methods with knowing the model structures. Since logic traps in existing evaluation methods can cause an inaccurate evaluation, we believe reducing the impact of these traps is the next question in the research field of post-hoc interpretations. In this section, we provide some thoughts for reducing the impact of logic trap 3: The change in attribution scores may be because the model reasoning process is changed rather than the attribution method is unreliable. To reduce the impact of this logic trap, we should try to guarantee the similarity in model reasoning when processing semantically similar inputs. In other words, we hope the target model used to test attribution methods more robustness to adversarial samples, which can be conducted through the following ways: 1 Enhancing the target model. The success of adversarial attacks on deep models motivates efforts to defend against such attacks. Thus, we can use these defense techniques, such as adversarial training (Tramr et al., 2018) and randomization (Xie et al., 2018), to enhance the target model and make it more robustness to adversarial samples. 2 Excluding predictions with low confidence. The deep model will give a prediction for a sample regardless of whether knowing how to deal with it. The randomness of reasoning increases with the uncertainty in model decisions (Bella et al., 2010). Thus, we can guarantee the stability of model reasoning by excluding low-confident predictions. For example, we can resorting to Confidence Calibration techniques (Guo et al., 2017; Seo et al., 2019), which calculate confidence interval for a predicted response. The proposed logic traps in existing evaluation methods have been ignored for a long time and negatively affected the research field. Though strictly accurate evaluation metrics for evaluating attribution methods might be a unicorn which will likely never be found, we should not just ignore these logic traps and draw conclusions recklessly. With a clear statement and awareness of these logic traps, we should reduce the focus on improving performance under such unreliable evaluation systems 5918 and shift it to reducing the impact of proposed logic traps. Moreover, other aspects of the research field should give rise to more attention, such as the applications of attribution scores (denoising data, improving the model performance, etc.) and proposing new explanation forms. This work is supported by the National Key Research and Development Program of China (No.2020AAA0106400), the National Natural Science Foundation of China (No.61922085, No.61976211, No.61906196). This work is also supported by the Key Research Program of the Chinese Academy of Sciences (Grant NO. ZDBS-SSW-JSC006), the independent research project of National Laboratory of Pattern Recognition and in part by the Youth Innovation Promotion Association CAS.
abstain abstain abstain abstain abstain objective objective abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain result other objective objective result abstain objective abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method abstain method abstain abstain abstain method method abstain result objective abstain other other