Pre-trained language models have recently shown that training on large corpora using the language modeling objective enables few-shot and zero-shot capabilities on a variety of NLP tasks, including commonsense reasoning tasks. This is achieved using text interactions with the model, usually by posing the task as a natural language text completion problem. While using language model probabilities to obtain task specific scores has been generally useful, it often requires task-specific heuristics such as length normalization, or probability calibration. In this work, we consider the question answering format, where we need to choose from a set of (free-form) textual choices of unspecified lengths given a context. We present ALC (Answer-Level Calibration), where our main suggestion is to model context-independent biases in terms of the probability of a choice without the associated context and to subsequently remove it using an unsupervised estimate of similarity with the full context. We show that our unsupervised answer-level calibration consistently improves over or is competitive with baselines using standard evaluation metrics on a variety of tasks including commonsense reasoning tasks. Further, we show that popular datasets potentially favor models biased towards easy cues which are available independent of the context. We analyze such biases using an associated F1-score. Our analysis indicates that answer-level calibration is able to remove such biases and leads to a more robust measure of model capability. Language models (LM), trained on large corpora, have been shown to exhibit few-shot and zero-shot learning capability (Radford et al., 2019; Brown et al., 2020) using only text interactions, as opposed to finetuning the model parameters using task specific training examples. Relying purely on text interactions for few-shot ability shifts the focus to designing and utilizing suitable task-specific natural language templates. In this work, we focus on free-form multiple choice question answering (and commonsense reasoning tasks in particular), where given a context and a set of choices of unspecified lengths, a model is required to select the most suitable choice. To enable zero-shot learning, the typical approach is to form textual sequences by concatenating the context independently with each choice and then scoring the concatenated strings using a pre-trained LM. While LM probabilities have been shown to provide useful estimates of choice probabilities given a context, there is no incentive to treat the choices as equal in the absence of the associated context. For example, the LM probabilities in a neutral context are likely to be determined by frequency. In this work, we explore the role of biases that are likely to be associated with the choices naturally due to the language modeling objective. We propose ALC 1 (Answer-Level Calibration), where we use a neutral context to model such biases and remove them using a scaling factor determined by how similarly a model handles the question context as compared to a neutral context. Further, we show that popular datasets favor models which rely on easy cues which are context independent. We use a bias-specific F1 score to analyze such biases. Our results indicate the need for answer-level calibration for more accurate estimates of model capabilities, or equivalently the design of better datasets. We hope our work will be useful for further research in both those directions. Specifically, we analyze context-independent biases related to length, part-of-speech (POS) and neutral context probabilities of the choices. 1. We present ALC, a model-agnostic approach to improve the unsupervised performance of pretrained LMs for free-form multiple choice question answering, including commonsense reasoning tasks. 2. We show that popular datasets favor models relying on context-independent easy cues and demonstrate the need for answer-level calibration to better estimate model capabilities. Prompts Jiang et al. (2020) show that manually created templates can be sub-optimal in extracting knowledge from LMs, and propose mining and paraphrasing-based approaches using training examples. Schick and Schtze (2021) highlight the importance of selecting templates for enabling few-shot learning. Calibration Probabilities output by neural networks are known to suffer from lack of calibration (Guo et al., 2017), including LM output probabilities (Braverman et al., 2020). Zhao et al. (2021) use token-level calibration to improve on few-shot classification and generation tasks. In contrast, we show that answer-level calibration is more suitable for the multiple choice setting that we consider. While we focus on free-form multiple choice questions in this work, when the choices are single tokens, for example in a classification task where the choices are True and False , answer-level calibration would behave similar to token-level calibration. As a result, answer-level calibration can be seen to have a more general scope as also illustrated empirically through our experiments. Further, our analysis (Section 3.4) shows that answer-level calibration provides a more reliable measure of model performance on datasets with potential biases. Finally, Jiang et al. (2021) explore supervised methods, including finetuning as well as post-hoc methods, to improve calibration using training examples. In this work, we focus mainly on unsupervised calibration. Answer-level calibration Brown et al. (2020) generally perform length normalization over the token probabilities for a choice, while observing that for a select few tasks they obtain performance gains when using an answer-level calibration scheme (which corresponds to the unscaled version in Equation 3 of ALC). They use task specific development sets to choose between length normalization and answer-level calibration which is undesirable for few-shot learning (Kann et al., 2019), and specifi-cally for zero-shot learning. In this work, we show that unscaled calibration (as in Equation 3) is suboptimal, compared to our proposed scaled version. More recently, Holtzman et al. (2021) also arrive at a formulation equivalent to the unscaled version of ALC but are motivated differently. Specifically, they hypothesize that the possibility of different surface forms of the same concept causes a competition between surface forms when scored by the LM. In contrast, we are motivated by calibration concerns and the presence of context-independent biases. We justify this motivation through bias associated evaluation (Section 5.2) for both the unscaled and scaled versions of ALC. One way to make the probability estimates of the choices more accurate is to enhance the context using more task-specific cues. For example, Brown et al. (2020) show that with just a few in-context examples, significant gains in performance can be obtained. At the same time, it has been shown that the order of examples as well as token-level calibration in such prompts can be critical for getting good performance (Zhao et al., 2021; Kumar and Talukdar, 2021). While the gains from enhanced context through additional examples may be complementary to answer-level calibration, we focus on the zero-shot setting in this work. In the zero-shot setting, Shwartz et al. (2020), working on the question answering format, propose generating textual clarifi-cations using the pre-trained LM itself, to enhance the context and improve zero-shot performance of pre-trained LM on commonsense reasoning tasks. While their method has a much higher computational cost, we use it as an unsupervised baseline and show improvement over it on most tasks we consider. We introduce the problem setting and notation in Section 3.1. We briefly describe our motivation in Section 3.2 and discuss the core idea of removing context-independent biases in Section 3.3. We provide the natural language formatting used in our experiments in Section A.3. We discuss bias associated measures in Section 3.4. We consider a problem setting where an example consists of a textual context C and K textual choices (or options) O k , k  [ K ] , and we need to predict which choice O k fits best in context C . For example, in the case of question answering, this amounts to answering a question contained in the context C . Additionally, we define an instance-independent neutral context C  , where we expect all choices to be equally likely. Denoting the gold answer by Y , the evaluation data is comprised of N instances defined by the tuples ( C i , [ O ik ] , Y i ) , k  [ K ] , i  [ N ] . Our main motivation is to evaluate the suitability of pretrained LMs for free-form multiple choice question answering where we contend that raw conditional phrase probabilities do not satisfy a natural requirement for such tasks (Equation 2). We suggest and evaluate modifications to meet this requirement. We aim to obtain a probabilistic model M which provides estimates PM ( O | C ) , the probability of a choice O given the context C . Predictions y for an example can subsequently be made using: y = argmax k ( PM ( O k | C )) (1) We wish to build such a model using a pretrained LM, e.g., GPT2. Such a LM, trained on the task of next word prediction, is expected to provide estimates of word probabilities given a textual context. For example, given the sequence of words w 1 w 2 ...w i , we expect GPT2 to provide probability estimates PL ( w i +1 | w 1 w 2 ...w i ) . Applying chain rule, we can obtain estimates of phrases given a textual context. For example, we could obtain estimates of PL ( O | C ) . Can PL ( O | C ) serve as a proxy for PM ( O | C ) ? It is tempting to expect the LM probabilities PL ( O | C ) to serve as a proxy for PM ( O | C ) when we can format the task in natural language. However, under the assumption that all choices O k are equally likely given a neutral context C  , this approximation can be sub-optimal. For it to be optimal, we would need PL ( O 1 | C  ) = PL ( O 2 | C  ) ... = PL ( OK | C  ) (2) However, given that these are task and instance specific choices, there is no incentive in the language modeling objective to ensure this condition. To address this, we define a new score SL ( O | C ) to behave as expected with a neutral context: SL ( O k | C ) = log PL ( O k | C )  log PL ( O k | C  ) (3) Predictions can subsequently be made using: y (cid:48) = argmax k ( SL ( O k | C )) (4) Scaling the bias term: Equation 3, while desirable, makes a strong assumption about how the bias is present in the LM. While valid unquestionably for the neutral context, the bias in a trained (on task-specific data, or on a task-independent pretraining corpus) model is likely to depend on the context as well. For instance, a longer or more familiar context (in terms of similarity to training contexts) may mean the model is less reliant on context-independent cues. We therefore define a scaled version for removing biases, where the function g outputs the scaling term (ranging in [0 , 1] ): S (cid:48) L ( O k | C ) = log PL ( O k | C )  g ( C, C  )  log PL ( O k | C  ) (5) We would want this formulation to preserve the requirement in Equation 2 which was satisfied by the unscaled version in Equation 3. Specifically, we want g ( C  , C  ) = 1 which would assign an equal score to each choice O k given a neutral context. To get a model-agnostic 2 estimate of g , we think of log PL ( O k | C ) and log PL ( O k | C  ) as outputs from different models M and M  respectively, and g as a measure of similarity between the models. Note that while M uses the available context C , M  uses only the neutral context C  . The intuition is that if M and M  are identical, there is no new information provided by M and we want to set g ( C, C  ) = 1 , leading to S (cid:48) L ( O k | C ) = 0 . On the other hand, if M and M  are very dissimilar, we can rely on the contextual scores of M and set g ( C, C  ) = 0 , leading to S (cid:48) L ( O k | C ) = log PL ( O k | C ) . Specifically, to estimate g , we compute a similarity metric between the token probabilities (across the model's entire vocabulary) output by the two models. where p fL indicates the probability vector output by the model across the vocabulary for the first token given the corresponding context. In this work, we consider Total Variation Distance (TVD), and Bhattacharyya Coefficient (BC) (Bhat-tacharyya, 1943). When using TVD, we subtract it from 1, to obtain a similarity estimate: g TVD ( C, C  ) = 1  0 . while we directly use BC: Consider an instance and choice specific attribute A i ( O ik ) which can take values a j , j  [ J ] . If we expect the attribute to be uncorrelated with task performance, we expect a model to perform similarly when evaluating subsets with different distributions of attributes A i ( O k ) = a j . If a model relies on specific values of the attribute and if the evaluation data has sufficient representation of that value, standard evaluation metrics which ignore this attribute may provide an erroneous estimate of the model capability. As an extreme example, consider A ( . ) to denote whether the selected choice corresponds to the shortest choice among all choice O k , k  [ K ] , with the attribute values being true/false. Assume then that the evaluation data is dominated by instances where A i ( Y i ) = true, i.e., with a high probability, the correct answer in the evaluation data is the shortest choice. Consider also a model which always chooses the shortest choice, irrespective of the content. The model would return close to perfect scores using standard evaluation metrics such as accuracy against gold labels. To analyze the impact of such attributes, we use a macro F1 score which takes into account the partitions created by an attribute. Recalling that an instance is represented by the tuple ( C i , [ O i k ] , Y i ) , i  [ N ] , and letting  Y i be the model prediction, we define precision (P), recall (R) and F1 scores for each attribute value a j , and subsequently an attribute specific macro F1 score ( F1 A ). P ( A,a j ) = # { ( A i (  Y i ) = a j ) & (  Y i = Y i ) } # { A i (  Y i ) = a j } (9) R ( A,a j ) = # { ( A i (  Y i ) = a j ) & (  Y i = Y i ) } # { A i ( Y i ) = a j } (10) F1 ( A,a j ) = 2  P ( A,a j )  R ( A,a j ) P ( A,a j ) + R ( A,a j ) (11) F1 A = Average( { F1 ( A,a j ) } ) (12) where # { . } denotes the count of the corresponding set. If the model performs similarly irrespective of the attribute value, the macro F1 score F1 A is equal to the standard measure of accuracy: Accuracy = # {  Y i = Y i } N (13) 4 Experimental Setup The datasets used and the corresponding prompts are described in Section 4.1. The LMs used are described in Section 4.2 and the baseline approaches in Section 4.3. Experimental results and analyses are presented in Section 5. We used a series of commonsense reasoning tasks and evaluated on the publicly available development sets. We used the same versions of the data as Shwartz et al. (2020) to allow for a direct comparison  COPA (Gordon et al., 2012), CommonsenseQA (Talmor et al., 2019), MCTACO (Zhou et al., 2019), SocialIQA (Sap et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2020). We also report on the adversarially generated large-scale SWAG dataset (Zellers et al., 2018). Further, we report on the AI2 Reasoning Challenge (ARC) (Clark et al., 2018), which has Easy and Challenge versions. As a representative dialog understanding task, we report on the DREAM (Sun et al., 2019) dataset. Finally, we report on a recent benchmark introduced for measuring multitask accuracy of pretrained models (referred to as Hendrycks in the following) Hendrycks et al. (2020). is associated with only one correct choice. For COPA, we also report on the test split due to the small size of COPA dev set. The sizes of the datasets used are reported in Appendix Table 8. All datasets contain questions in English language. We briefly describe these datasets in Section A.2. Examples for each dataset along with contextual ( C ) and neutral ( C  ) prompts used in this work are captured in Section A.3. We experiment with GPT2 (Radford et al., 2019) variants distilgpt2, gpt-small, gpt-medium, gpt2-large and gpt2-xl. The size of models used is reported in Appendix Table 9. While the gpt-* models have been trained similarly as described in Radford et al. (2019), distilgpt2 has been pretrained with the supervision of GPT2 3 (Wolf et al., 2020). For most of our experiments, we utilize the gpt2-xl model. Please refer Section A.1 for additional details about the experimental setup. Uncalibrated : Predictions are made using uncalibrated probabilities from a LM, log PL ( O | C ) , computed as the sum of conditional log-probabilities output by the model for the tokens in O . Length normalized : Predictions are made using length-normalized probabilities from a LM, log PL ( O | C ) , computed as the mean of conditional log-probabilities output by the model for the tokens in O . Model ARC DREAM SWAG Hendrycks-test Easy Challenge Humanities STEM Social sciences Other Token calibration 35.09 20.40 40.20 29.53 23.38 22.70 25.45 25.51 Uncalibrated 58.25 27.76 48.14 49.30 26.99 24.16 31.52 31.55 Length normalized 50.70 29.43 48.77 65.36 29.33 26.47 30.84 32.85 ALC Unscaled 53.33 33.11 52.99 57.04 31.05 29.13 32.76 35.26 TVD 60.00 29.43 52.50 53.77 28.80 25.98 32.24 33.07 BC 56.49 33.78 53.14 59.16 30.31 27.60 32.60 34.58 Self-talk : We use the official code repository 4 of self-talk (Shwartz et al., 2020) using gpt2-xl as both the scoring model and the knowledge source. Token calibration: Following Zhao et al. (2021), we use the probability vector output, p fN by the model at the first token given the neutral context to calibrate the model probabilities. Specifically, each token probability p is offset by p fN and renormalized: p (cid:48) = softmax( p  p fN ) . We also tried an alternative variant suggested by Zhao et al. (2021) where p (cid:48) = softmax( p/p fN ) but this generally did worse and we skip the corresponding results. We aim to answer the following questions: Q1 How does ALC compare with baselines using standard evaluation (accuracy) on free-form multiple choice question answering tasks? (Section 5.1) Q2 Does the aforementioned evaluation reflect true model capability? To answer this question, we perform a series of bias associated evaluations (see Section 3.4) and also evaluate whether ALC helps overcome such biases. Specifically, we evaluate on biases related to answer length, POS tag and context-ignorant LM probability. (Section 5.2) Q3 Does ALC improve expected calibration error (Guo et al., 2017)? (Section 5.3) 5.1 Standard Evaluation The overall results for the commonsense reasoning tasks (considered by Shwartz et al. (2020)) using standard evaluation of ALC, as well as the base-670 Model POS = noun POS = verb POS = adj P R F1 P R F1 P R F1 Commonsenseqa Size 902 149 142 Uncalibrated 36.93 39.47 38.16 39.57 36.91 38.19 40.48 23.94 30.09 Length normalized 35.07 31.37 33.12 33 44.97 38.07 33.33 33.8 33.57 ALC (Unscaled) 48.68 45.01 46.77 43.75 56.38 49.27 52.74 54.23 53.47 ALC (BC) 49.32 48.34 48.82 48.84 56.38 52.34 59.32 49.3 53.85 Table 4: POS bias analysis: We consider subsets of data using the POS tag of the first token and report on P, R and F1 scores (lowest values are underlined). lines, with gpt2-xl are presented in Table 1 (top). We also report on an unscaled ablation of ALC. Note that ALC outperforms the uncalibrated baseline on all datasets except WinoGrande (where all models perform poorly and we drop it from further discussions). Further, the significant gains compared to token calibration (which generally does worse than the uncalibrated baseline) show that answer-level calibration is more suited for unsupervised commonsense question answering when there is no constraint on the lengths of candidate choices. Finally, ALC outperforms or is competitive with self-talk 5 while being significantly less computationally intensive. ALC requires scoring two strings (context input and neutral input) for each choice, while self-talk requires generating hundreds of clarification texts using data-dependent templates and subsequently scoring them. We also report on the average gain over the uncalibrated baseline across gpt2 models of varying sizes (Table 9) in Table 1 (middle) and observe similar trends as in the case of gpt2-xl. While our focus is zero-shot unsupervised evaluation, we also perform few-shot (1-shot and 4-shot) evaluation In general, for k-shot evaluation, we sample 100 sets of size k from an unseen split 6 of the dataset. A few-shot context is obtained by concatenating training examples with a newline token. We report the average performance on the evaluation set in Table 1 (bottom) and observe similar trends as before. We present the standard zero-shot evaluation on additional datasets in Table 2. The trends are sim-5 Please see Section A.4 for a note explaining the unusually high relative performance of baselines on some tasks when compared to self-talk. 6 For few-shot evaluation, we sample from the training split for all except COPA and MCTACO datasets where we sample from the dev set and report on the test set. ilar except for the SWAG (see Section 5.2 for an explanation) and the Hendrycks datasets (see Table 11). Finally, while our focus is causal language models, we also present results using RoBERTa-large (a masked language model) in Table 10. Again, we observe similar trends. In the subsequent sections, we show that the evaluation using the accuracy metric may not reveal true model capabilities as the datasets may favor models which utilize easy cues for predicting the answer. Next, to gain a better understanding of the model capabilities, we analyze the performance associated with undesirable biases related to length, POS tag and context-ignorant LM probability. Specifically, we define the following attributes (see Section 3.4): Shortest Attribute A i ( O ik ) is set to true if O ik is the shortest (number of tokens) choice among the choices O ik (cid:48) , k (cid:48)  [ K ] . Otherwise, the attribute is set to false. Longest Defined similar to Shortest, but set to true if O ik is the longest answer and false otherwise. POS Attribute A i ( O ik ) is set to the POS tag of the first token in the choice O ik . We don't consider POS tags which occur less than a threshold (25) in the evaluation data. LM-Best Attribute A i ( O ik ) is set to true if O ik is the most likely choice using context-ignore (neu-tral input) LM probability. Otherwise, it is set to false. LM-Worst Defined similar to LM-Best, but set to true when O ik is the least likely choice and false otherwise. Finally, we consider length-normalized versions of LM-Best and LM-Worst, referred to as LM-671 Model LM-Best = true LM-Worst = True P R F1 P R F1 PIQA Size 1195 643 Uncalibrated 70.53 94.31 80.7 71.67 26.75 38.96 Length normalized 73.96 86.28 79.64 63.06 43.55 51.52 ALC (Unscaled) 77.42 54.23 63.78 45.35 70.61 55.23 ALC (BC) 75.74 81 78.29 59.46 51.79 55.36 ARC (Easy) Size 183 109 Baseline 52.96 83.06 64.68 72.22 23.85 35.86 Length normalized 60.56 59.56 60.06 39.47 41.28 40.36 ALC (Unscaled) 81.71 36.61 50.57 36.2 73.39 48.48 ALC (BC) 64.15 55.74 59.65 45.04 54.13 49.17 ARC (Challenge) Size 64 86 Uncalibrated 23.78 68.75 35.34 28.57 4.65 8 Length normalized 27.55 42.19 33.33 29.03 20.93 24.32 ALC (Unscaled) 38.71 18.75 25.26 33.87 48.84 40 ALC (BC) 29.76 39.06 33.78 36.99 31.4 33.96 Table 5: Context-ignorant LM bias analysis: We consider subsets of data where the correct choice corresponds to the best/worst choice as per the context-ignorant (neutral context) LM probability and report on P, R and F1 scores (lowest values are underlined). Norm-Best and LM-Norm-Worst respectively. Briefly, our experiments reveal that while the datasets considered don't share a similar bias pattern, each usually suffers from at least one bias considered in this work, i.e., there is a drop in performance when measured using the bias associated score. We present the detailed results for commonsense reasoning tasks in Appendix Table 12, using gpt2-xl model, while highlighting the key takeaways here. Recall that in the absence of biases in the model, the F1 score should match the accuracy score. In the following sections, we provide a more directed analysis on the presence of such biases, on datasets where such biases are most prominent, and if ALC helps alleviate such biases. We create subsets of the CommonsenseQA and SocialIQA dev set with specific properties to evaluate if the LM-Baseline has the associated biases and if they are addressed by ALC. First, we create subsets of examples where the shortest/longest answer is the correct answer. We expect longer sentences to have lower probabilities than shorter sentences with the uncalibrated baseline. Additionally, with the length normalized variant, where the final score is obtained as the mean of conditional log-probabilities instead of the sum (as in the uncalibrated baseline), longer sentences could potentially be favored. We report the uncalibrated and ALC's performance in Table 3. Note that both uncalibrated baseline and the length-normalized variants favor one subset at the cost of the other, while ALC improves on both. In particular, the uncalibrated baseline has a much poorer recall when the longest answer is correct. On the other hand, the length normalized variant has a much poorer recall when the shortest answer is correct. The results indicate that ALC provides a viable alternative to length normalization for handling length biases. We analyze potential part of speech (POS) tag biases in Table 4. Considering the CommonsenseQA dataset, we create subsets of the data where the correct answer is of the POS tag noun, verb or adjective. Note that ALC shows less variation in performance (F1) across these subsets when compared to uncalibrated baseline while improving on 672 Model PIQA SWAG LM-Norm-Best Baseline 65.38 49.38 Length normalized 60.16 60.97 ALC (Unscaled) 58.89 58.12 ALC (BC) 67.92 59.48 LM-Norm-Worst Baseline 65.38 39.90 Length normalized 60.16 46.99 ALC (Unscaled) 58.89 49.48 ALC (BC) 67.92 51.14 Table 6: Context-ignorant normalized scores: LM-Norm-Best (top) and LM-Norm-Worst (bottom) macro F1 evaluation on SWAG and PIQA datasets (lowest values are underlined and highest values are in bold). each subset. In particular, the maximum difference in F1 scores is 8.1 for the uncalibrated baseline while it is 5.03 for ALC (BC). ALC also improves over the length normalized variant for each subset. To understand how much of the unsupervised performance comes from context-independent LM biases, we analyze subsets where the correct answer is most/least likely without the context. We report the performance on the PIQA and ARC datasets in Table 5 and show that such biases indeed exist. The key takeaway is that the standard evaluation metrics may not give an accurate estimate of performance and that ALC provides more reliable estimates. Finally, we report macro F1 scores for LM-Norm-Best and LM-Norm-Worst evaluation in Table 6 on PIQA and SWAG datasets. The results indicate that the datasets favours length normalization aware scoring irrespective of the context. When we measure the bias associated score, ALC generally performs better. Given a score S ( O k | C ) for each choice O k , we can compute a confidence estimate conf( O k | C ) as: Guo et al. (2017) compute expected calibration error (ECE) by partitioning N confidence predictions into R equal bins B r , r  [1 , R ] and computing the weighted average of the absolute difference between the confidence and accuracy in each bin: ECE = R (cid:88) r =1 | B r | N | acc( B r )  conf( B r ) | (15) where acc() and conf() measure the accuracy and mean confidence respectively in a bin. We set the number of bins to be 20. We report the average difference in accuracy and ECE compared to the uncalibrated baseline across the evaluation datasets (except WinoGrande) in Table 7. When compared to the uncalibrated baseline, ALC provides gains in calibration error while also improving performance. Length-normalization also improves ECE, presumably by correcting for length bias. However, length-normalization does not improve performance on an average. The relative performance gains of ALC can be explained through the handling of additional biases beyond length bias. We propose ALC (Answer-Level Calibration), an unsupervised method to improve performance of pretrained language models. We show that, when compared to existing baselines, ALC is more suitable for free-form multiple choice question answering, including commonsense reasoning tasks. We also show that popular datasets favor models which rely on easy cues for predictions, and that ALC provides more reliable estimates of model capabilities by getting rid of some of these biases.
abstain abstain abstain method method result result method method abstain abstain method abstain abstain abstain objective objective result method result abstain method result objective other other other other objective method abstain abstain other method other other objective other other objective objective abstain abstain abstain method abstain method method method method method method abstain method abstain method method method method abstain method method method abstain method abstain method method abstain abstain method method method abstain method method method abstain method method method abstain method abstain abstain abstain abstain abstain method method other abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result result result abstain result abstain abstain abstain abstain abstain abstain abstain other abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective result result