Understanding how linguistic structure is encoded in contextualized embedding could help explain their impressive performance across NLP. Existing approaches for probing them usually call for training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation's goodness. In this work, we argue that doing so can be unreliable because different representations may need different classifiers. We develop a heuristic, DIRECTPROBE , that directly studies the geometry of a representation by building upon the notion of a version space for a task. Experiments with several linguistic tasks and contextualized embeddings show that, even without training classifiers, DIRECTPROBE can shine light into how an embedding space represents labels, and also anticipate classifier performance for the representation. Distributed representations of words (e.g., Peters et al., 2018; Devlin et al., 2019) have propelled the state-of-the-art across NLP to new heights. Recently, there is much interest in probing these opaque representations to understand the information they bear (e.g., Kovaleva et al., 2019; Conneau et al., 2018; Jawahar et al., 2019). The most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy (e.g. Kassner and Schtze, 2020; Perone et al., 2018; Yaghoobzadeh et al., 2019; Krasnowska-Kieras and Wrblewska, 2019; Wallace et al., 2019; Pruksachatkun et al., 2020). Using these classifiers, criteria such as accuracy or model complexity are used to evaluate the representation quality for the task (e.g. Goodwin et al., 2020; Pimentel et al., 2020a; Michael et al., 2020). task. However, their ability to reveal the information in a representation is occluded by numerous factors, such as the choice of the optimizer and the initialization used to train the classifiers. For example, in our experiments using the task of preposition supersense prediction (Schneider et al., 2018), we found that the accuracies across different training runs of the same classifier can vary by as much as  8% ! (Detailed results can be found in Appendix F.) Indeed, the very choice of a classifier influences our estimate of the quality of a representation. For example, one representation may achieve the best classification accuracy with a linear model, whereas another may demand a multi-layer per-ceptron for its non-linear decision boundaries. Of course, enumerating every possible classifier for a task is untenable. A common compromise involves using linear classifiers to probe representations (Alain and Bengio, 2017; Kulmizev et al., 2020), but doing so may mischaracterize representations that need non-linear separators. Some work recognizes this problem (Hewitt and Liang, 2019) and proposes to report probing results for at least logistic regression and a multi-layer percep-tron (Eger et al., 2019), or to compare the learning curves between multiple controls (Talmor et al., 2020). However, the success of these methods still depends on the choices of classifiers. In this paper, we pose the question: Can we evaluate the quality of a representation for an NLP task directly without relying on classifiers as a proxy? Our approach is driven by a characterization of not one, but all decision boundaries in a representation that are consistent with a training set for a task. This set of consistent (or approximately consistent) classifiers constitutes the version space for the task (Mitchell, 1982), and includes both simple (e.g., linear) and complex (e.g., non-linear) classifiers for the task. However, perfectly characterizing the version space for a problem presents computational challenges. To develop an approximation, we note that any decision boundary partitions the underlying feature space into contiguous regions associated with labels. We present a heuristic approach called DIRECTPROBE , which builds upon hierarchical clustering to identify such regions for a given task and embedding. The resulting partitions allow us to directly probe the embeddings via their geometric properties. For example, distances between these regions correlate with the difficulty of learning with the representation: larger distances between regions of different labels indicates that there are more consistent separators between them, and imply easier learning, and better generalization of classifiers. Further, by assigning test points to their closest partitions, we have a parameter-free classifier as a side effect, which can help benchmark representations without committing to a specific family of classifiers (e.g., linear) as probes. Our experiments study five different NLP tasks that involve syntactic and semantic phenomena. We show that our approach allows us to ascertain, without training a classifier, (a) if a representation admits a linear separator for a dataset, (b) how different layers of BERT differ in their representations for a task, (c) which labels for a task are more confusable, (d) the expected performance of the best classifier for the task, and (e) the impact of fine-tuning. In summary, the contributions of this work are: 1. We point out that training classifiers as probes is not reliable, and instead, we should directly analyze the structure of a representation space. 2. We formalize the problem of evaluating representations via the notion of version spaces and introduce DIRECTPROBE , a heuristic method to approximate it directly which does not involve training classifiers. 3. Via experiments, we show that our approach can help identify how good a given representation will be for a prediction task. 1 2 Representations and Learning In this section, we will first briefly review the relationship between representations and model learning. Then, we will introduce the notion of (cid:15) -version spaces to characterize representation quality. The problem of training a predictor for a task can be divided into two sub-problems: (a) representing data, and (b) learning a predictor (Bengio et al., 2013). The former involves transforming input objects x words, pairs of words, sentences, etc. into a representation E ( x )  R n that provides features for the latter. Model learning builds a classifier h over E ( x ) to make a prediction, denoted by the function composition h ( E ( x )) . Figure 1 illustrates the two sub-tasks and their roles in figuratively transporting an input x towards a prediction with a probability of error below a small (cid:15) . For the representation E 1 , the best classifier h 1 falls short of this error requirement. The representation E 4 does not need an additional classifier because it is identical to the label. The representations E 2 and E 3 both admit classifiers h 2 and h 3 that meet the error threshold. Further, note, that E 3 leaves less work for the classifier than E 2 , suggesting that it is a better representation as far as this task is concerned. The quality of a representation E for a task is a function of both the performance and the complexity of the best classifier h over that representation. Two observations follow from the above discussion. First, we cannot enumerate every possible classifier to find the best one. Other recent work, such 2 In addition to performance and complexity of the best classifier, other aspects such as sample complexity, the stability of learning, etc are also important. We do not consider them in this work: these aspects are related to optimization and learnability, and more closely tied to classifier-based probes. as that of Xia et al. (2020) make a similar point. Instead, we need to resort to an approximation to evaluate representations. Second, trivially, the best representation for a task is identical to an accurate classifier; in the illustration in Figure 1, this is represented by E 4 . However, such a representation is over-specialized to one task. In contrast, learned representations like BERT promise task-independent representations that support accurate classifiers. Given a classification task, we seek to disentangle the evaluation of a representation E from the classifiers h that are trained over it. To do so, the first step is to characterize all classifiers supported by a representation. Classifiers are trained to find a hypothesis (i.e., a classifier) that is consistent with a training set. A representation E admits a set of such hypotheses, and a learner chooses one of them. Consider the top-left example of Figure 2. There are many classifiers that separate the two classes; the figure shows two linear ( h 1 and h 2 ) and one non-linear ( h 3 ) example. Given a set H of classifiers of interest, the subset of classifiers that are consistent with a given dataset represents the version space with respect to H (Mitchell, 1982). To account for errors or noise in data, we define an (cid:15) -version space: the set of hypothesis that can achieve less than (cid:15) error on a given dataset . Let us formalize this definition. Suppose H represents the whole hypothesis space consisting of all possible classifiers h of interest. The (cid:15) -version space V (cid:15) ( H , E, D ) expressed by a representation E for a labeled dataset D is defined as: V (cid:15) ( H , E, D ) (cid:44) { h  H | err ( h, E, D )  (cid:15) } (1) where err represents training error. Note that the (cid:15) -version space V (cid:15) ( H , E, D ) is only a set of functions and does not involve any learning. However, understanding a representation requires examining its (cid:15) -version spacea larger one would allow for easier learning. In previous work, the quality of a representation E for a task represented by a dataset D is measured via properties of a specific h  V (cid:15) ( H , E, D ) , typically a linear probe. Commonly measured properties include generalization error (Kim et al., 2019), minimum description length of labels (Voita and Titov, 2020) and complexity (Whitney et al., 2020). Instead, we seek to directly evaluate the (cid:15) -version space of a representation for a task, without committing to a restricted set of probe classifiers. Although the notion of an (cid:15) -version space is well defined, finding it for a given representation and task is impossible because it involves enumerating all possible classifiers. In this section, we will describe a heuristic to approximate the (cid:15) -version space. We call this approach as DIRECTPROBE . Each classifier in V (cid:15) ( H , E, D ) is a decision boundary in the representation space that separates examples with different labels (see Figure 2, left). The decisions of a classifier can be mimicked by a set of piecewise linear functions. Figure 2 (middle) shows two examples. At the top is the simple case with linearly separable points. At the bottom is a more complicated case with a circular separator. The set of the piecewise linear function that matches its decisions needs at least three lines. The ideal piecewise linear separator partitions training points into groups, each of which contains points with exactly one label. These groups can be seen as defining convex regions in the embedding space (see Figure 2, left). Any classifier in V (cid:15) ( H , E, D ) must cross the regions between the groups with different labels; these are the regions that separate labels from each other, as shown in the gray areas in Figure 2 (middle). Inspired by this, we posit that these regions between groups with different labels, and indeed the partitions themselves, offer insight into V (cid:15) ( H , E, D ) . Although finding the set of all decision boundaries remain hard, finding the regions between convex groups that these piecewise linear functions splits the data into is less so. Grouping data points in this fashion is related to a well-studied problem, namely clustering, and several recent works have looked at clustering of contextualized representations (Reimers et al., 2019; Aharoni and Goldberg, 2020; Gupta et al., 2020). In this work, we have a new clustering problem with the following criteria: (i) All points in a group have the same label. We need to ensure we are mimicking the decision boundaries. (ii) There are no overlaps between the convex hulls of each group. If convex hulls of two groups do not overlap, there must exist a line that can separates them, as guaranteed by the hyperplane separation theorem. (iii) Minimize the number of total groups. Otherwise, a simple solution is that each data point becomes a group by itself. Note that the criteria do not require all points of one label to be grouped together. For example, in Figure 2 (bottom right), points of the circle (i.e.,  ) class are in three subgroups. To summarize what we have so far: we transformed the problem of finding the (cid:15) -version space into a clustering problem with specific criteria. Next, let us see a heuristic for partitioning the training set into clusters based on these criteria. To find clusters as described in 3.2, we define a simple bottom-up heuristic clustering strategy that forms the basis of DIRECTPROBE (Algorithm 1). In the beginning, each example x i with label y i in a training set D is a cluster C i by itself. In each iteration, we select the closest pair of clusters with the same label and merge them (lines 4, 5). If the convex hull of the new cluster does not overlap with all other clusters 3 , we keep this new cluster (line 9). Otherwise, we flag this pair (line 7) and 3 Note that all other clusters here means the other clusters with different labels. There is no need to prohibit overlap between clusters with the same label since they might be merged in the next iterations. We define the distance between clusters C i and C j as the Euclidean distance between their centroids. Although Algorithm 1 does not guarantee the minimization criterion in 3.2 since it is greedy heuristic, we will see in our experiments that, in practice, it works well. Overlaps Between Convex Hulls A key point of Algorithm 1 is checking if the convex hulls of two sets of points overlap (line 6). Suppose we have two sets C = { x C 1 , . . . , x Cn } and C (cid:48) = { x C (cid:48) 1 , . . . , x C (cid:48) m } . We can restate this as the problem of checking if there is some vector w  (cid:60) n and a number b  (cid:60) such that:  x Ci  C, w (cid:62) E ( x Ci ) + b  1 ,  x C (cid:48) j  C (cid:48) , w (cid:62) E ( x C (cid:48) j ) + b   1 . where E is the representation under investigation. We can state this problem as a linear program that checks for feasibility of the system of inequalities. If the LP problem is feasible, there must exist a separator between the two sets of points, and they do not overlap. In our implementation, we use the Gurobi optimizer (Gurobi Optimization, 2020) to solve the linear programs. 4 3.4 Noise: Controlling (cid:15) Clusters with only a small number of points could be treated as noise. Geometrically, a point in the neighborhood of other points with different labels 4 Algorithm 1 can be made faster by avoiding unnecessary calls to the solver. Appendix A gives a detailed description of these techniques, which are also incorporated in the code release. could be thought of as noise. Other clusters can not merge with it because of the no-overlap constraint. As a result, such clusters will have only a few points (one or two in practice). If we want zero error rate on the training data, we can keep these noise points; if we allow a small error rate (cid:15) , then we can remove these noise clusters. In our experiments, for simplicity, we keep all clusters. Before looking at the analysis offered by the partitions obtained via DIRECTPROBE in 5, let us first enumerate the English NLP tasks and representations we will encounter. Our main experiments focus on BERT base,cased , and we also show additional analysis on other contextual representations: ELMo (Peters et al., 2018) 5 , BERT large,cased (Devlin et al., 2019), RoBERTa base and RoBERTa large (Liu et al., 2019b). We refer the reader to the Appendix C for further details about these embeddings. We use the average of subword embeddings as the token vector for the representations that use subwords. We use the original implementation of ELMo, and the HuggingFace library (Wolf et al., 2020) for the others. We conduct our experiments on five NLP tasks that cover the varied usages of word representations (token-based, span-based, and token pairs) and include both syntactic and semantic prediction problems. The Appendix D has more details about the tasks to help with replication. Preposition supersense disambiguation represents a pair of tasks, involving classifying a preposi-tion's semantic role (SS-role) and semantic function (SS-func) . Following the previous work (Liu et al., 2019a), we only use the single-token prepositions in the Streusle v4.2 corpus (Schneider et al., 2018). Part-of-speech tagging (POS) is a token level prediction task. We use the English portion of the parallel universal dependencies treebank (ud-pud Nivre et al., 2016). We use the dataset of semeval2010 task 8 (Hen-drickx et al., 2010). To represent the pair of nominals, we concatenate their embeddings. Some nominals could be spans instead of individual tokens, and we represent them via the average embedding of the tokens in the span. Dependency relation (DEP) is the task of predicting the syntactic dependency relation between a token w head and its modifier w mod . We use the universal dependency annotation of the English web treebank (Bies et al., 2012). As with semantic relations, to represent the pair of tokens, we concatenate their embeddings. The key starting point of this work is that restricting ourselves to linear probes may be insufficient. To validate the results of our analysis, we evaluate a large collection of classifiersfrom simple linear classifiers to two-layers neural networksfor each task. For each one, we choose the best hyper-parameters using cross-validation. From these classifiers, we find the best test accuracy of each task and representation. All classifiers are trained with the scikit-learn library (Pedregosa et al., 2011). To reduce the impact of randomness, we trained each classifier 10 times with different initializations, and report their average accuracy. The Appendix E summarizes the best classifiers we found and their performance. DIRECTPROBE helps partition an embedding space for a task, and thus characterize its (cid:15) -version space. Here, we will see that these clusters do indeed characterize various linguistic properties of the representations we consider. linear separability of representations for a task. The best scenario is when the number of clusters equals the number of labels. In this case, examples with the same label are placed close enough by the representation to form a cluster that is separable from other clusters. A simple linear multi-class classifier can fit well in this scenario. In contrast, if the number of clusters is more than the number of labels, then some labels are distributed across multiple clusters (as in Figure 2, bottom). There must be a non-linear decision boundary. Consequently, Embedding Linear SVM #Clusters Training Accuracy BERT base,cased 100 17 BERT large,cased 100 17 RoBERTa base 100 17 RoBERTa large 99.97 1487 ELMo 100 17 Table 1: Linearity experiments on POS tagging task. In other words, using the clusters, and without training a classifier, we can answer the question: can a linear classifier fit the training set for a task with a given representation? To validate our predictions, we use the training accuracy of a linear SVM (Chang and Lin, 2011) classifier. If a linear SVM can perfectly fit ( 100% accuracy) a training set, then there exist linear decision boundaries that separate the labels. Table 1 shows the linearity experiments on the POS task, which has 17 labels in total. All representations except RoBERTa large have 17 clusters, suggesting a linearly separable space, which is confirmed by the SVM accuracy. We conjecture that this may be the reason why linear models usually work for BERT-family models. Of course, linear separability does not mean the task is easy or that the best classifier is a linear one. We found that, while most representations we considered are linearly separable for most of our tasks, the best classifier is not always linear. We refer the reader to Appendix E for the full results. As we mentioned in 3.1, a learning process seeks to find a decision boundary that separates clusters with different labels. Intuitively, a larger gap between them would make it easier for a learner to find a suitable hypothesis h that generalizes better. We use the distance between convex hulls of clusters as an indicator of the size of these gaps. We note that the problem of computing the distance between convex hulls of clusters is equivalent to finding the maximum margin separator between them. To find the distance between two clusters, we train a linear SVM (Chang and Lin, 2011) that Figure 3: Here we juxtapose the minimum distances between clusters and the best classifier accuracy for all 12 layers. separates them and compute its margin. The distance we seek is twice the margin. For a given representation, we are interested in the minimum distance across all pairs of clusters with different labels. Higher layers usually have larger (cid:15) -version spaces. Different layers of BERT play different roles when encoding liguistic information (Tenney et al., 2019). To investigate the geometry of different layers of BERT, we apply DIRECTPROBE to each layer of BERT base,cased for all five tasks. Then, we computed the minimum distances among all pairs of clusters with different labels. By comparing the minimum distances of different layers, we answer the question: how do different layers of BERT differ in their representations for a task? Figure 3 shows the results on all tasks. In each subplot, the horizontal axis is the layer index. For each layer, the blue circles (left vertical axis) is the best classifier accuracy, and the red triangles (right vertical axis) is the minimum distance described above. We observe that both best classifier accuracy and minimum distance show similar trends across different layers: first increasing, then decreasing. It shows that minimum distance correlates with the best performance for an embedding space, though it is not a simple linear relation. Another interesting observation is the decreasing performance and minimum distance of higher layers, which is also corroborated by Ethayarajh (2019) and Liu et al. (2019a). Fine-tuning expands the (cid:15) -version space. Past work (Peters et al., 2019; Arase and Tsujii, 2019; Merchant et al., 2020) has shown that fine-tuning pre-trained models on a specific task improves performance, and fine-tuning is now the de facto procedure for using contextualized embeddings. In this experiment, we try to understand why fine-tuning can improve performance. Without training classifiers, we answer the question: What changes in the embedding space after fine-tuning? We conduct the experiments described in 5.2.1 on the last layer of BERT base,cased before and after fine-tuning for all tasks. Table 2 shows the results. We see that after fine-tuning, both the best classifier accuracy and minimum distance show a big boost. It means that fine-tuning pushes the clusters away from each other in the representation space, which results in a larger (cid:15) -version space. As we discussed in 5.2, a larger (cid:15) -version space admits more good classifiers and allows for better generalization. Small distances between clusters can confuse a classifier. By comparing the distances between clusters, we can answer the question: Which labels for a task are more confusable? We compute the distances between all the pairs of labels based on the last layer of BERT base,cased . 6 Based on an even split of the distances, we partition all label pairs into three bins: small, medium, and large. For each task, we use the predictions of the best classifier to compute the number of mis-6 For all tasks, BERT base,cased space (last layer) is linearly separable. So, the number of label pairs equals the number of cluster pairs. classified label pairs for each bin. For example, if the clusters associated with the part of speech tags ADV and ADJ are close to each other, and the best classifier misclassified ADV as ADJ, we put this error pair into the bin of small distance. The distribution of all errors is shown in Table 3. This table shows that a large majority of the misclassified labels are concentrated in the small distance bin. For example, in the supersense role task (SS-role), 97 . 17% of the errors happened in small distance bin. The number of label pairs of each bin is shown in the parentheses. Table 3 shows that small distances between clusters indeed confuse a classifier and we can detect it without training classifiers. We can predict the expected performance of the best classifier. Any h  V (cid:15) ( H , E, D ) is a predictor for the task D on the representation E . As a by-product of the clusters from DIRECTPROBE , we can define a predictor. The prediction strategy is simple: for a test example, we assign it to its closest cluster. 7 Indeed, if the label of the cluster is the true label of the test point, then we know that there exists some classifier that allows this example to be correctly labeled. We can verify the label every test point and compute the aggregate accuracy to serve as an indicator of the generalization ability of the representation at hand. We call this accuracy the intra-accuracy . In other words, without training classifiers, we can answer the question: given a representation, what is the expected performance of the best classifier for a task? Figure 4 compares the best classifier accuracy, and the intra-accuracy of the last layer of different embeddings. Because our assignment strategy is similar to nearest neighbor classification (1-kNN), which assigns the unlabelled test point to its closest labeled point, the figure also compares to the 1-kNN accuracy. First, we observe that intra-accuracy always outperforms the simple 1-kNN classifier, showing that DIRECTPROBE can use more information from the representation space. Second, we see that the intra-accuracy is close to the best accuracy for some tasks (Supersense tasks and POS tagging). Moreover, all the pearson correlation coefficients between best accuracy and intra-accuracy (showed in the parentheses alongside each task title) suggest a high linear correlation between best classifier accuracy and intra-accuracy. That is, the intra-accuracy can be a good predictor of the best classifier accuracy for a representation. From this, we argue that intra-accuracy can be interpreted as a benchmark accuracy of a given representation without actually training classifiers. The distances between a test point and all the clusters from the training set can not only be used to predict the label but also can be used to identify difficult examples as per a given representation. Doing so could lead to re-annotation of the data, and perhaps lead to cleaner data, or to improved embeddings. Using the supersense role task, we show a randomly chosen example of a mismatch 7 To find the distance between the convex hull of a cluster and a test point, we find a max-margin separating hyperplane by training a linear SVM that separates the point from the cluster. The distance is twice the distance between the hyperplane and the test point. The data labels the word our as GESTALT , while the embedding places it in the neighborhood of POSSESSOR . The annotation guidelines for these labels (Schneider et al., 2017) notes that GESTALT is a supercategory of POSSESSOR . The latter is specifically used to identify cases where the possessed item is alienable and has monetary value. From this definition, we see that though the annotated label is GESTALT , it could arguably also be a POSSESSOR if phone numbers are construed as alienable possessions that have monetary value. Importantly, it is unclear whether BERT base,cased makes this distinction. Other examples we examined required similarly nuanced analysis. This example shows DIRECTPROBE can be used to identify examples in datasets that are potentially mislabeled, or at least, require further discussion. In addition to the classifier based probes described in the rest of the paper, a complementary line of work focuses on probing the representations using a behavior-based methodology. Controlled test sets (Senel et al., 2018; Jastrzebski et al., 2017) are designed and errors are analyzed to reverse-engineer what information can be encoded by the model (e.g., Marvin and Linzen, 2018; Ravichander et al., 2021; Wu et al., 2020). Another line of work probes the space by opening up the representation space or the model (e.g., Michel et al., 2019; Voita et al., 2019). There are some efforts to inspect the space from a geometric perspective (e.g., Ethayarajh, 2019; Mimno and Thompson, 2017). Our work extends this line of work to connect the geometric structure of embedding space with classifier performance without actually training a classifier. Recent work (Pimentel et al., 2020b; Voita and Titov, 2020; Zhu and Rudzicz, 2020) probe representations from an information theoretic perspective. These efforts still need a probability distribution p ( y | x ) from a trained classifier. In 5.3, we use clusters to predict labels. In the same vein, the conditional probability p ( y | x ) can be obtained by treating the negative distances between the test point x and all clusters as predicted scores and normalizing via softmax. Our formalization can fit into the information theoretic analysis and yet avoid training a classifier. Our analysis and experiments open new directions for further research: Novel pre-training target: The analysis presented here informs us that larger distance between clusters can improve classifiers. This could guide loss function design when pre-training representations. Quality of a representation: In this paper, we focus on the accuracy of a representation. We could seek to measure other properties (e.g., complexity) or proxies for them. These analytical approaches can be applied to the (cid:15) -version space to further analyze the quality of the representation space. Theory of representation: Learning theory, e.g. VC-theory (Vapnik, 2013), describes the learnabil-ity of classifiers; representation learning lacks of such theoretical analysis. The ideas explored in this work ( (cid:15) -version spaces, distances between clusters being critical) could serve as a foundation for an analogous theory of representations. In this work, we ask the question: what makes a representation good for a task? We answer it by developing DIRECTPROBE , a heuristic approach builds upon hierarchical clustering to approximate the (cid:15) -version space. Via experiments with several contextualized embeddings and linguistic tasks, we showed that DIRECTPROBE can help us understand the geometry of the embedding space and ascertain when a representation can successfully be employed for a task. We thank the members of the Utah NLP group and Nathan Schneider for discussions and valuable insights, and reviewers for their helpful feedback. We also thank the support of NSF grants #1801446 (SATC) and #1822877 (Cyberlearning).
abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective method result abstain abstain method result abstain abstain abstain abstain abstain objective objective result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain other other other other objective other other method other method objective other method method other other objective abstain objective result other other