Probing has become an important tool for analyzing representations in Natural Language Processing (NLP). For graphical NLP tasks such as dependency parsing, linear probes are currently limited to extracting undirected or unlabeled parse trees which do not capture the full task. This work introduces DEPPROBE , a linear probe which can extract labeled and directed dependency parse trees from embeddings while using fewer parameters and compute than prior methods. Leveraging its full task coverage and lightweight parametrization, we investigate its predictive power for selecting the best transfer language for training a full biaffine attention parser. Across 13 languages, our proposed method identifies the best source treebank 94% of the time, outperforming competitive baselines and prior work. Finally, we analyze the informativeness of task-specific subspaces in contextual embeddings as well as which benefits a full parser's non-linear parametrization provides. Pre-trained, contextualized embeddings have been found to encapsulate information relevant to various syntactic and semantic tasks out-of-the-box (Tenney et al., 2019; Hewitt and Manning, 2019). Quantifying this latent information has become the task of probes  models which take frozen embeddings as input and are parametrized as lightly as possible (e.g. linear transformations). Recent proposals for edge probing (Tenney et al., 2019) and structural probing (Hewitt and Manning, 2019) have enabled analyses beyond classification tasks, including graphical tasks such as dependency parsing. They are able to extract dependency graphs from embeddings, however these are either undirected (Hewitt and Manning, 2019; Hall Maudslay et al., 2020) or unlabeled (Kulmizev et al., 2020), thereby capturing only a subset of the full task. In this work, we investigate whether this gap can be filled and ask: Can we construct a lightweight probe which can produce fully directed and labeled dependency trees? Using these trees, we further aim to study the less examined problem of transferability estimation for graphical tasks, extending recent work targeting classification and regression tasks (Nguyen et al., 2020; You et al., 2021). Specifically: How well do our probe's predictions correlate with the transfer performance of a full parser across a diverse set of languages? To answer these questions, we contribute DEPPROBE (Figure 1), the first linear probe to extract directed and labeled dependency trees while using fewer parameters than prior work and three orders of magnitude fewer trainable parameters than a full parser (Section 3). As this allows us to measure labeled attachment scores (LAS), we investigate the degree to which our probe is predictive of cross-lingual transfer performance of a full parser across 13 typologically diverse languages, finding that our approach chooses the best transfer language 94% of the time, outperforming competitive baselines and prior work (Section 4). Finally, we perform an in-depth analysis of which latent information is most relevant for dependency parsing as well as which edges and relations benefit most from the expressivity of the full parser (Section 5). 1 1 Code available at https://personads.me/x/acl-2022-code. Given the ubiquitous use of contextualized embeddings (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021), practitioners have turned to various methods for analyzing their linguistic features (Rogers et al., 2020). Hewitt and Manning (2019) examine these intrinsic properties in greater detail for English dependency parsing using a structural probe , finding that undirected dependency graphs are recoverable from BERT by learning a linear transformation on its embeddings (Section 3.1). Extending the structural probe of Hewitt and Manning (2019) to 12 languages, Chi et al. (2020) extract undirected dependency graphs from mBERT (Devlin et al., 2019), further showing that head-to-child difference vectors in the learned subspace cluster into relations from the Universal Dependencies taxonomy (de Marneffe et al., 2014). Building on both the structural and tree depth probes (Hewitt and Manning, 2019), Kulmizev et al. (2020) extract directed dependency graphs from mBERT for 13 languages (Section 3.2). Further variations to structural probing include regularization of the linear transformation (Limisiewicz and Marecek, 2021) as well as alternative objective functions (Hall Maudslay et al., 2020). None of the proposed linear probing approaches so far are able to produce full dependency parse trees (i.e. directed and labeled), however the closer a probe approximates the full task, the better it quantifies relevant information (Hall Maudslay et al., 2020). It would for example be desirable to estimate LAS for parsing a target treebank with a model trained on a different source without having to train a resource-intensive parser (e.g. Dozat and Manning, 2017) on each source candidate. Although performance prediction methods for such scenarios exist, they typically do not cover graph prediction (Nguyen et al., 2020; You et al., 2021). In order to bridge the gap between full parsers and unlabeled probes, in addition to the gap between full fine-tuning and lightweight performance prediction, this work proposes a linear probe which can extract labeled and directed dependency parse trees while using less compute than prior methods (Section 3). We use our probe's LAS to evaluate its predictive power for full parser performance and leverage its linear nature to investigate how dependencies are represented in subspaces of contextual embeddings (Section 5). In order to construct a directed and labeled dependency parse tree for a sentence s consisting of the words { w 0 , . . . , w N } , we require information on the presence or absence of edges between words, the directionality of these edges (  w i , w j ) , and the relationships { r 0 , . . . , r N } which they represent. Using the contextualized embeddings { h 0 , . . . , h N } with h i  R e , prior probing work has focused on the first step of identifying edges (Section 3.1) and later directionality (Section 3.2). In this work, we propose a probe which completes the final relational step (Section 3.3) and simultaneously provides a more efficient method for identifying directionality (Section 3.4). The structural probe introduced by Hewitt and Manning (2019) recovers the first piece of information (i.e. the undirected graph) remarkably well. Here, the probe is a linear transformation B  R e  b with b < e which maps contextual embeddings into a subspace in which the distance measure d B ( h i , h j ) = (cid:113) ( B h i  B h j ) T ( B h i  B h j ) (1) between h i and h j is optimized towards the distance between two words in the dependency graph d P ( w i , w j ) , i.e. the number of edges between the words. For each sentence, the loss is defined as the mean absolute difference across all word pairs: LB ( s ) = 1 N 2 N (cid:88) i =0 N (cid:88) j =0 (cid:12)(cid:12) d P ( w i , w j )  d B ( h i , h j ) (cid:12)(cid:12) . In order to extract an undirected dependency graph, one computes the distances for a sentence's word pairs using d B and extracts the minimum spanning tree (Jarnk, 1930; Prim, 1957; MST). Apart from the structural probe B , Hewitt and Manning (2019) also probe for tree depth. Using another matrix C  R e  c , a subspace is learned in which the squared L 2 norm of a transformed embedding (cid:107) C h i (cid:107) 22 corresponds to a word's depth in the tree, i.e. the number of edges from the root. Kulmizev et al. (2020) combine the structural and tree depth probe to extract directed graphs. This directed probe (DIRPROBE ) constructs a score matrix M  RN  N for which each entry corresponds to a word pair's negative structural distance  d B ( h i , h j ) . The shallowest node in the depth subspace C is set as root. Entries in M which correspond to an edge between w i and w j for which the word depths follow (cid:107) C h i (cid:107) 22 > (cid:107) C h j (cid:107) 22 are set to  . A word's depth in subspace C therefore corresponds to edge directionality. The directed graph is built from M using Chu-Liu-Edmonds decoding (Chu and Liu, 1965; Edmonds, 1967). DIRPROBE extracts directed dependency parse trees, however it would require additional complexity to label each edge with a relation (e.g. using an additional probe). In the following, we propose a probe which can extract both directionality and relations while using fewer parameters and no dynamic programming-based graph-decoding algorithm. The incoming edge of each word w i is governed by a single relation. As such the task of dependency relation classification with l relations can be simpli-fied to a labeling task using a linear transformation L  R e  l for which the probability of a word's relation r i being of class l k is given by: p ( r i = l k | w i ) = softmax ( L h i ) k (3) and optimization uses standard cross-entropy loss given the gold label r  i for each word w i : LL ( s ) =  1 NN (cid:88) i =0 ln p ( r  i | w i ) . Should dependency relations be encoded in contextualized embeddings, each dimension of the subspace L will correspond to the prevalence of information relevant to each relation, quantifiable using relation classification accuracy (RelAcc). Combining structural probing (Section 3.1) and dependency relation probing (Section 3.3), we propose a new probe for extracting fully directed and labeled dependency trees (DEPPROBE ). It combines undirected graphs and relational information in a computationally efficient manner, adding labels while requiring less parameters than prior unlabeled or multi-layer-perceptron-based approaches. As outlined in Algorithm 1 and illustrated in Figure 1, DEPPROBE uses the distance matrix DB Algorithm 1: DEPPROBE Inference 1 input Distance matrix DB  RN  N , p ( l k | w i ) of relation label l k given w i 2 w r  argmax w i p ( root | w i ) 3 T w  { w r } , T e  {} 4 while |T w | < N do 5 w i , w j  argmin w i ,w j DB ( w i  T w , w j ) 6 r j  argmax l k p ( l k | w j ) with l k (cid:54) = root 7 T w  T w  { w j } 8 T e  T e  { (  w i , w j , r j ) } 9 end 10 return T e derived from the structural probe B in conjunction with the relation probabilities of the relational probe L (line 1). The graph is first rooted using the word w r for which p ( root | w r ) is highest (line 2). Iterating over the remaining words until all w j are covered in T w , an edge is drawn to each word w j from its head w i based on the minimum distance in DB . The relation r j for an edge (  w i , w j , r j ) is determined by taking the relation label l k which maximizes p ( r j = l k | w j ) with l k (cid:54) = root (line 6). The edge is then added to the set of labeled tree edges T e . With edge directionality being inferred as simply pointing away from the root, this procedure produces a dependency graph that is both directed and labeled without the need for additional complexity, running in O ( n 2 ) while dynamic programming-based decoding such as DIRPROBE have runtimes of up to O ( n 3 ) (Stanojevic and Cohen, 2021). Constructing dependency trees from untuned embeddings requires the matrices B and L , totaling e  b + e  l trainable parameters. Optimization can be performed using gradient descent on the sum of losses LB + LL . With l = 37 relations in UD, this constitutes a substantially reduced training effort compared to prior probing approaches (with subspace dimensionalities b and c typically set to 128) and multiple magnitudes fewer fine-tuned parameters than for a full biaffine attention parser. Parsers In our experiments, we use the deep biaffine attention parser (BAP) by Dozat and Manning (2017) as implemented in van der Goot et al. (2021) as an upper bound for MLM-based pars-7713 ing performance. As it is closest to our work, we further reimplement DIRPROBE (Kulmizev et al., 2020) with b = 128 and c = 128. Note that this approach produces directed, but unlabeled dependency graphs. Finally, we compare both methods to our directed and labeled probing approach, DEPPROBE with b = 128 and l = 37. All methods use mBERT (Devlin et al., 2019) as their encoder ( e = 768). For BAP, training the model includes fine-tuning the encoder's parameters, while for both probes they remain fixed and only the linear transformations are adjusted. This results in 183M tuned parameters for BAP, 197k for DIRPROBE and 127k for DEPPROBE . Hyper-parameters are set to the values reported by the authors, 2 while for DEPPROBE we perform an initial tuning step in Section 4.2. Target Treebanks As targets, we use the set of 13 treebanks proposed by Kulmizev et al. (2019), using versions from Universal Dependencies v2.8 (Zeman et al., 2021). They are diverse with respect to language family, morphological complexity and script (Appendix A). This set further includes EN-EWT (Silveira et al., 2014) which has been used in prior probing work for hyperparameter tuning, allowing us to tune DEPPROBE on the same data. Metrics We report labeled attachment scores (LAS) wherever possible (BAP, DEPPROBE ) and unlabeled attachment scores (UAS) for all methods. For DEPPROBE 's hyperparameters, we evaluate undirected, unlabeled attachment scores (UUAS) as well as relation classification accuracy (RelAcc). One notable difference to prior work is that we include punctuation both during training and evaluation  contrary to prior probing work which excludes all punctuation (Hewitt and Manning, 2019; Kulmizev et al., 2020; Hall Maudslay et al., 2020)  since we are interested in the full parsing task. Training Each method is trained on each target treebank's training split and is evaluated on the test split. For cross-lingual transfer, models trained on one language are evaluated on the test splits of all other languages without any further tuning. For DEPPROBE tuning (Section 4.2) we use the development split of EN-EWT. BAP uses the training schedule implemented in van der Goot et al. (2021) while DIRPROBE and 2 For better comparability, we use the best single layer reported by Kulmizev et al. (2020) instead of the weighted sum over all layers. DEPPROBE use AdamW (Loshchilov and Hutter, 2019) with a learning rate of 10  3 which is reduced by a factor of 10 each time the loss plateaus (see also Hewitt and Manning, 2019). Both probing methods are implemented using PyTorch (Paszke et al., 2019) and use mBERT as implemented in the Transformers library (Wolf et al., 2020). Each model is trained with three random initializations of which we report the mean. As prior work has repeatedly found that MLM layers encode different linguistic information, the layers which are most relevant for a probe's task are typically first identified (Tenney et al., 2019; Hewitt and Manning, 2019). Following this paradigm, we train DEPPROBE on embeddings from each layer of mBERT. Layer 0 is equivalent to the first, non-contextualized embeddings while layer 12 is the output of the last attention heads. The probe is trained on EN-EWT and evaluated on its development split using UUAS for the structural transformation B (akin to Hewitt and Manning, 2019) as well as RelAcc for the relational transformation L . Figure 2 shows that structure is most prevalent around layer 6 at 78 UUAS, corroborating the 68 range identified by prior work (Tenney et al., 2019; Hewitt and Manning, 2019; Chi et al., 2020). Dependency relations are easiest to retrieve at around layer 7 with an accuracy of 86%. The standard deviation across initializations is around 0.1 in both cases. Based on these tuning results, we use layer 6 for structural probing and layer 7 for relational probing in the following experiments. Figure 3 lists UAS for all methods and LAS for BAP and DEPPROBE both on target-language test data (=L) and zero-shot transfer targets (  L). Table 3c further shows the mean results for each setting. Unsurprisingly, the full parametrization of BAP performs best, with in-language scores of 88 LAS and 91 UAS. For zero-shot transfer, these scores drop to 35 LAS and 52 UAS, with some language pairs seeing differences of up to 85 points: e.g. JA  JA (93 LAS) versus AR  JA (8 LAS) in Figure 3a. This again confirms the importance of selecting appropriate source data for any given target. Both probes, with their limited parametrization, fall short of the full parser's performance, but still reach up to 73 LAS and 79 UAS. DIRPROBE has a mean in-language UAS which is 3 points higher than for DEPPROBE , attributable to the more complex decoder. Due to DIRPROBE 's output structures being unlabeled, we cannot compare LAS. DEPPROBE reaches a competitive 67 UAS despite its much simpler decoding procedure and appears to be more stable for zero-shot transfer as it outperforms DIRPROBE by around 2 UAS while maintaining a lower standard deviation. Most importantly, it produces directed and labeled parses such that we can fully compare it to BAP. Considering that DEPPROBE has more than three orders of magnitude fewer tunable parameters, a mean in-language LAS of 60 is considerable and highlights the large degree of latent dependency information in untuned, contextual embeddings. For zero-shot transfer, the performance gap to BAP narrows to 13 LAS and 14 UAS. Given that DEPPROBE provides a highly parameter-efficient method for producing directed, labeled parse trees, we next investigate whether its performance patterns are indicative of the full parser's performance and could aid in selecting an appropriate source treebank for a given target without having to train the 183 million parameters of BAP. Setup Comparing UAS and LAS of BAP with respective scores of DEPPROBE and DIRPROBE , we compute the Pearson correlation coefficient  and the weighted Kendall's  w (Vigna, 2015). The latter can be interpreted as corresponding to a cor-7715 MODELLAS UAS   w   w L2V .86 .72 .80 .70 DIRPROBE   .91 .81 DEPPROBE .97 .88 .94 .85 Table 1: Transfer Correlation with BAP. relation in [  1 , 1] , and that given a probe ranking one source treebank over another, the probability of this higher rank corresponding to higher performance in the full parser is  w +12 . All reported correlations are significant at p < 0 . 001 . Similarly, differences between correlation coefficients are also significant at p < 0 . 001 as measured using a standard Z-test. In addition to the probes, we also compare against a method commonly employed by practitioners by using the cosine similarity of typological features from the URIEL database as represented in lang2vec (Littell et al., 2017; L2V) between our 13 targets (details in Appendix A). Results Table 1 shows that the L2V baseline correlates with final parser performance, but that actual dependency parses yield significantly higher correlation and predictive power. For UAS, we find that despite having similar attachment scores, DEPPROBE performance correlates higher with BAP than that of DIRPROBE , both with respect to predicting the ability to parse any particular language as well as ranking the best source to transfer from. Using the labeled parse trees of DEPPROBE results in almost perfect correlation with BAP's LAS at  = .97 as well as a  w of .88, highlighting the importance of modeling the full task and including dependency relation information. Using Kendall's  w with respect to LAS, we can estimate that selecting the highest performing source treebank from DEPPROBE to train the full parser will be the best choice 94% of the time for any treebank pair. Why does DEPPROBE predict transfer performance more accurately than DIRPROBE despite its simpler architecture? As each probe consists only of two matrices optimized to extract tree structural, depth MODELLAS UAS   w   w SSA-STRUCT .68 .42 .60 .43 SSA-DEPTH .62 .34 .53 .35 SSA-REL .73 .55 .65 .53 Table 2: SSA Correlation with BAP. or relational information, we can directly compare the similarity of all task-relevant parameters across languages against the full BAP's cross-lingual performance. In order to measure the similarity of probe matrices from different languages, we use mean subspace angles (Knyazev and Argentati, 2002; SSA), similarly to prior probing work (Chi et al., 2020). Intuitively, SSA quantifies the energy required to transform one matrix to another by converting the singular values of the transformation into angles between 0  and 90  . SSAs are computed for the structural probe (SSA-STRUCT ) which is equivalent in both methods, DIRPROBE 's depth probe (SSADEPTH ) and DEPPROBE 's relational probe (SSAREL ). We use Pearson  and the weighted Kendall's  w to measure the correlation between cross-lingual probe SSAs and BAP performance. This allows us to investigate which type of information is most important for final parsing performance. From Table 2, we can observe that SSAs between probes of different languages correlate less with transfer performance than UAS or LAS (Ta-ble 1), underlining the importance of extracting full parses. Among the different types of dependency information, we observe that SSAs between the relational probes used by DEPPROBE correlate highest with final performance at .73 for LAS and .65 for UAS. Structural probing correlates significantly both with BAP's LAS and UAS at .68 and .60 respectively, but to a lesser degree. Probes for tree depth have the lowest correlation at .62 for LAS and .53 for UAS. Despite tree depth being a distinctive syntactic feature for language pairs such as the agglutinative Turkish and the more function word-based English, depth is either not as relevant for BAP or may be represented less consistently in embeddings across languages, leading to lower correlation between SSAs and final performance. In the following analysis we investigate performance differences between the full BAP and DEPPROBE across all 13 targets in order to identify finer-grained limitations of the linear approach and also which kinds of dependencies benefit from full parameter tuning and non-linear decoding. Edge Length Figure 5 shows offsets between gold and predicted head positions. The majority of heads are predicted correctly with a ratio of 92.1% for BAP and 69.7% for DEPPROBE . Both methods are less accurate in predicting long-distance edges with length 150250, resulting in offsets of ca. 100 (aggregated into < and > in Figure 5). Most likely, this is due to these edges' overall sparsity in the data (only 6.7% of edges cover a distance of more than 10 tokens) as well as their higher overall subjective difficulty. Nonetheless, BAP is able to capture such dependencies more accurately as shown by its lower error rates for long edges compared to those of DEPPROBE . In addition to very distant head nodes, BAP also seems to recover more of the nuanced edges in the [  5 , 5] interval. This range is particularly impactful for downstream performance as the edges in our target treebanks have a median length of 2 (mean length 3.62 with  = 5.70). The structural probing loss (Equation 2) and the simple linear parametrization of the probe are able to capture a large number of these edges as evidenced by overall low error rates, but lack the necessary expressivity in order to accurately capture all cases. Relations Looking at RelAcc for each category in the UD taxonomy (de Marneffe et al., 2014) in Figure 4 allows us to identify where higher parametrization and more complex decoding are required for high parsing performance. While we again observe that performance on all relations is higher for BAP than for DEPPROBE , a large subset of the relations is characterized by comparable or equivalent performance. These include simple punctuation ( punct ), but also the majority of function word relations such as aux , case , clf , det and mark as well as coordination (e.g. cc , conj ). We attribute the high performance of DEPPROBE on these relations to the fact that the words used to express them typically stem from closed classes and consequently similar embeddings: e.g., determiners the/a/an (EN), case markers di/da (IT). open class words are also captured by the linear probe. These include the modifiers advmod , amod and discourse as well as some nominal relations such as expl , nmod , nsubj and nummod . As prior work has identified PoS information in untuned embeddings (Tenney et al., 2019), the modifiers are likely benefiting from the same embedding features. The fact that DEPPROBE nonetheless identifies syntax-specific relations such as nsubj , and to a lesser degree obj and obl , indicates the presence of context-dependent syntactic information in addition to PoS. The larger the set of possible words for a relation, the more difficult it is to capture with the probe. The functional cop (copula) relation provides an informative example: In English (and related languages), it is almost exclusively assigned to the verb be resulting in 85% RelAcc, while in non-European languages such as Japanese it can be ascribed to a larger set which often overlaps with other relations (e.g. aux ) resulting in 65% RelAcc. BAP adapts to each language by tuning all parameters while DEPPROBE , using fixed embeddings, reaches competitive scores on European languages, but performs worse in non-European settings (details in Appendix B). Besides capturing larger variation in surface forms, BAP also appears to benefit from higher expressivity when labeling clausal relations such as ccomp , csubj . These relations are often characterized not only by surface form variation, but also by PoS variation of head/child words and overlap with other relation types (e.g. clausal subjects stem from verbs or adjectives), making them difficult to distinguish in untuned embeddings. Simultaneously, they often span longer edges compared to determiners or other function words. Another relation of particular importance is root as it determines the direction of all edges predicted by DEPPROBE . An analysis of the 14% RelAcc difference to BAP reveals that both methods most frequently confuse root with relations that fit the word's PoS, e.g. NOUN roots with nsubj or nmod . For the majority PoS VERB (70% of all root ), we further observe that DEPPROBE predicts twice as many xcomp and parataxis confusions compared to BAP, likely attributable to their root -similar function in subclauses. Since their distinction hinges on context, the full parser, which also tunes the contextual encoder, is better equipped to differentiate between them. The last category in which BAP outperforms DEPPROBE includes rare, treebank-specific relations such as reparandum (reference from a corrected word to an erroneous one). Again, the larger number of tunable parameters in addition to the non-linear decoding procedure of the full parser enable it to capture more edge cases while DEPPROBE 's linear approach can only approximate a local optimum for any relations which are represented non-linearly. Efficiency When using a probe for performance prediction, it is important to consider its computational efficiency over the full parser's fine-tuning procedure. In terms of tunable parameters, DEPPROBE has 36% fewer parameters than DIRPROBE and three orders of magnitude fewer parameters than BAP. In practice, this translates to training times in the order of minutes instead of hours. Despite its simple O ( n 2 ) decoding procedure compared to dynamic programming-based graph-decoding algorithms ( O ( n 3 ) ), DEPPROBE is able to extract full dependency trees which correlate highly with downstream performance while maintaining high efficiency (Section 4.4). With DEPPROBE , we have introduced a novel probing procedure to extract fully labeled and directed dependency trees from untuned, contextualized embeddings. Compared to prior approaches which extract structures lacking labels, edge directionality or both, our method retains a simple linear parametrization which is in fact more lightweight and does not require complex decoders (Section 3). To the best of our knowledge, this is the first linear probe which can be used to estimate LAS from untuned embeddings. Using this property, we evaluated the predictive power of DEPPROBE on cross-lingual parsing with respect to the transfer performance of a fully fine-tuned biaffine attention parser. Across the considered 169 language pairs, DEPPROBE is surprisingly effective: Its LAS correlates significantly ( p < 0 . 001 ) and most highly compared with unlabeled probes or competitive language feature baselines, choosing the best source treebank in 94% of all cases (Section 4). Leveraging the linearity of the probe to analyze structural and relational subspaces in mBERT embeddings, we find that dependency relation information is particularly important for parsing performance and cross-lingual transferability, compared to both tree depth and structure. DEPPROBE , which models structure and relations, is able to recover many functional and syntactic relations with competitive accuracy to the full BAP (Section 5). Finally, the substantially higher efficiency of DEPPROBE with respect to time and compute make it suitable for accurate parsing performance prediction. As contemporary performance prediction methods lack formulations for graphical tasks and handcrafted features such as lang2vec are not available in all transfer settings (e.g. document domains, MLM encoder choice), we see linear approaches such as DEPPROBE as a valuable alternative. We would like to thank the NLPnorth group for insightful discussions on this work  in particular Elisa Bassignana and Mike Zhang. Additional thanks to ITU's High-performance Computing Cluster team. Finally, we thank the anonymous reviewers for their helpful feedback. This research is supported by the Independent Research Fund Denmark (Danmarks Frie Forskningsfond; DFF) grant number 9063-00077B.
abstain abstain method objective objective method abstain abstain abstain abstain objective abstain abstain objective objective method other other other other other other other other other objective objective method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain method method abstain method method method method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method objective method abstain result abstain abstain result other other other other