A central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spuriousi.e., the model might not rely on it when making predictions. In this paper, we try to find an encoding that the model actually uses, introducing a usage-based probing setup. We first choose a behavioral task which cannot be solved without using the linguistic property. Then, we attempt to remove the property by intervening on the model's representations. We contend that, if an encoding is used by the model, its removal should harm the performance on the chosen behavioral task. As a case study, we focus on how BERT encodes grammatical number, and on how it uses this encoding to solve the number agreement task. Experimentally, we find that BERT relies on a linear encoding of grammatical number to produce the correct behavioral output. We also find that BERT uses a separate encoding of grammatical number for nouns and verbs. Finally, we identify in which layers information about grammatical number is transferred from a noun to its head verb. Pre-trained language models have enabled researchers to build models that achieve impressive performance on a wide array of natural language processing (NLP) tasks (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020). How these models encode and use the linguistic information necessary to perform these tasks, however, remains a mystery. Over recent years, a number of works have tried to demystify the inner workings of various pre-trained language models (Alain and Bengio, 2016; Adi et al., 2017; Elazar et al., 2021), but no comprehensive understanding of how the models work has emerged. Such analysis methods are typically termed probing , and are methodologically diverse. In our assessment, most research in probing can be taxonomized into three distinct paradigms. In the first paradigm, diagnostic probing , researchers typically train a supervised classifier to predict a linguistic property from the models' representations. High accuracy is then interpreted as an indication that the representations encode information about the property (Alain and Bengio, 2016; Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018). A second family of methods, behavioral probing , consists in observing a model's behavior directly, typically studying the model's predictions on hand-picked evaluation datasets (Linzen et al., 2016; Goldberg, 2019; Warstadt et al., 2020; Ettinger, 2020). Finally, causal probing methods rely on interventions to evaluate how specific components impact a model's predictions (Giulianelli et al., 2018; Vig et al., 2020b; Elazar et al., 2021). In this paper, we will investigate how linguistic properties are encoded in a model's representations, where we use the term encoding to mean the subspace on which a model relies to extractor decodethe information. While probing has been extensively used to investigate whether a linguistic property is encoded in a set of representations, it still cannot definitively answer whether a model actually uses a certain encoding. Diagnostic probes, for instance, may pick up on a spurious encoding of a linguistic property, i.e., an encoding that allows us to extract our target property from the representation, but which the model being probed may not actually use to make a prediction. Combining the three paradigms above, we instead seek to find encodings that are actually used by a pre-trained model, which we term functional encodings . To that end, we take a usage-based perspective on probing. Under this perspective, a researcher first identifies a linguistic property to investigate (e.g., grammatical number), and selects a behavioral task which requires knowledge of this 8818 property (e.g., selecting a verb's inflection which agrees in number with its subject). The researcher then performs a causal intervention with the goal of removing a specific encoding (of the linguistic property under consideration) from the model's representations. If the encoding is a functional encoding, i.e., an encoding that the model indeed uses to make a prediction, then the intervention should prevent the model from solving the task. 1 Finally, once a functional encoding is discovered, we can use it to track how the property's information flows through the model under investigation. As a case study, we examine how BERT (Devlin et al., 2019) uses grammatical number to solve a number agreement task. In English, grammatical number is a binary morpho-syntactic property: A word is plural or singular. In turn, subjectverb number agreement is a behavioral task; it inspects whether a model can predict the correct verbal inflection given its subject's number. For a model to solve the task, it thus requires information about the grammatical number of the subject and the verb. Our goal is to find how the model encodes this information when using it to make predictions. In other words, we want to find the structure from which the model decodes number information when solving the task. In our experiments, we make three findings. First, our experiments provide us with strong evidence that BERT relies on a linear functional encoding of grammatical number to solve the number agreement task. Second, we find that nouns and verbs do not have a shared functional encoding of number; in fact, BERT relies on disjoint sub-spaces to extract their information. Third, our usage-based perspective allows us to identify where number information (again, as used by our model to make predictions) is transferred from a noun to its head verb. Specifically, we find that this transfer occurs between BERT's 3 rd and 8 th layers, and that most of this information is passed indirectly through other tokens in the sentence. A variety of approaches to probing have been proposed in the literature. In this paper, we taxonomize them into three paradigms: (i) diagnostic probing, (ii) behavioral probing, and (iii) causal probing. Diagnostic Probing. Traditionally, probing papers focus on training supervised models on top of fixed pre-trained representations (Adi et al., 2017; Hall Maudslay et al., 2020). The general assump-tion behind the work is that, if a probe achieves high accuracy, then the property of interest is encoded in the representations. Many researchers have expressed a preference for linear classifiers in probing (Alain and Bengio, 2016; Ettinger et al., 2016; Hewitt and Manning, 2019), suggesting that a less complex classifier gives us more insight into the model. Others, however, called this criterion into question (Tenney et al., 2019b,a; Voita and Titov, 2020; Papadimitriou et al., 2021; Sinha et al., 2021; Pimentel et al., 2020a; Pimentel and Cotterell, 2021). Notably, Hewitt and Liang (2019) proposed that complex classifiers may learn to extract a property by themselves, and may thus not reflect any true pattern in the representations. Further, Pimentel et al. (2020b) showed that, under a weak assumption, contextual representations encode as much information as the original sentences. Ergo, it is not clear what we can conclude from diagnostic probing alone. Behavioral Probing. Another probing paradigm analyzes the behavior of pre-trained models on carefully curated datasets. By avoiding the use of diagnostic probes, they do not fall prey to the criticism abovetasks are directly performed by the model, and thus must reflect the pre-trained models' acuity. One notable example is Linzen et al. (2016), who evaluate a language model's syntactic ability via a careful analysis of a number agreement task. By controlling the evaluation, Linzen et al. could disentangle the model's syntactic knowledge from a heuristic based on linear ordering. In a similar vein, a host of recent work makes use of carefully designed test sets to perform behavioral analysis (Ribeiro et al., 2020; Warstadt et al., 2020; Warstadt and Bowman, 2020; Lovering et al., 2021; Newman et al., 2021). While behavioral probing often yields useful insights, the paradigm typically treats the model itself as a blackbox, thus failing to explain how individual components of the model work. Causal Probing. Finally, a third probing paradigm relies on causal interventions (Vig et al., 2020b; Tucker et al., 2021; Ravfogel et al., 2021). In short, the researcher performs causal interventions that modify parts of the network during a forward pass (e.g., a layer's hidden representations) 8819 to determine their function. For example, Vig et al. (2020a) fix a neuron's value while manipulating the model's input to evaluate this neuron's role in mediating gender bias. Relatedly, Elazar et al. (2021) propose a method to erase a target property from a model's intermediate layers. They then analyze the effect of such interventions on a masked language model's outputs. Under our usage-based perspective, our goal is to find a functional encodingi.e., an encoding that the model actually uses when making predictions. We achieve this by relying on a combination of the paradigms discussed in 2. To this end, we first need a behavioral task that requires the model to use information about the target property. We then perform a causal intervention to try to remove this property's encoding. We explain both these components in more detail now. Behavioral Task. We first require a behavioral task which can only be solved with information about the target property. The choice of task and target property are thus co-dependent. Further, we require our model to perform well on this task. On one hand, if the model cannot achieve high performance on the behavioral task, we cannot be sure the model encodes the target property, e.g., grammatical number, at all. On the other hand, if the model can perform the task, it must make use of the property. Causal Intervention. Our goal in this work is to answer a causal question: Can we identify a property's functional encoding? We thus require a way to intervene in the model's representations. If a model relies on an encoding to make predictions, removing it should harm the model's performance on the behavioral task. If follows that, by measuring the impact of our interventions on the model's behavioral output, we can assess whether our model was indeed decoding information from our targeted encoding. The empirical portion of this paper focuses on a study of how BERT encodes grammatical number in English. We choose number as our object of study because it is a well understood morpho-syntactic property in English. Thus, we are able to formulate simple hypotheses about how BERT passes information about number when performing number agreement. We use Linzen et al.'s (2016) number agreement task as our behavioral task. In English, a verb and its subject agree in grammatical number (Corbett, 2006). Consider, for instance, the sentences: (1) In the above sentences, both (1-a) and (1-c) are grammatical, but (1-b) and (1-d) are not; this is because, in the latter two sentences, the highlighted verb does not agree in number with its subject. The subjectverb number agreement task evaluates a model's ability to predict the correct verbal inflection, measuring its preference for the grammatical sentence. In this task, the probed model is typically asked to predict the verb's number given its context. The model is then considered successful if it assigns a larger probability to the correct verb inflection: context : The boy that holds the keys [MASK] to the movies. success : p ( goes | context ) > p ( go | context ) failure : p ( go | context ) > p ( goes | context ) In this setting, the subject is usually called the cue of the agreement, and the verb is called the target . Examples similar to the above are often designed to study the impact of distractors (the word keys in (1-c) and (1-d)) on the model's ability to predict the correct verb form. Success on the task is usually taken as evidence that a model is able to track syntactic dependencies. In this regard, this phenomena has been studied in a variety of settings to investigate the syntactic abilities of neural language models (Gulordava et al., 2018; Marvin and Linzen, 2018; Newman et al., 2021; Lasri et al., 2022). In this work, however, we do not use this task to make claims about the syntactic abilities of the model, as done by Linzen et al. (2016). Instead, we employ it as a case study to investigate how BERT encodes and uses grammatical number. A number of studies have investigated how grammatical number is encoded in neural language mod-8820 els. 2 Most of this work, however, focuses on diagnostic probes (Klafka and Ettinger, 2020; Torroba Hennigen et al., 2020). These studies are thus agnostic about whether the probed models actually use the encodings of number they discover. Some authors, however, do consider the relationship between how the model encodes grammatical number and its predictions. Notedly, Giulianelli et al. (2018) use a diagnostic probe to investigate how an LSTM encodes number in a subjectverb number agreement setting. Other approaches (Lakretz et al., 2019; Finlayson et al., 2021) have been proposed to apply interventions at the neuron level and track their effect on number agreement. In this work, we look for functional encodings of grammatical numberencodings which are in fact used by our probed model when solving the task. We discuss how to identify and remove an encoding from a set of contextual representations using diagnostic probing. Our use of diagnostic probing is thus twofold. For a model to rely on an encoding of our property when making predictions, the property must be encoded in its representations. We thus first use diagnostic probing to measure the amount of information a representation contains about the target linguistic property. In this sense, diagnostic probing serves to sanity-check our experimentsif we cannot extract information from the representations, there is no point in going forward with our analysis. Second, we make use of diagnostic probing in the context of amnesic probing (Elazar et al., 2021), which allows us to determine whether this probe finds a functional or a spurious encoding of the target property. In this section, we discuss how to estimate the amount of extractable number information in our probed model's representations. This is the probing perspective taken by Pimentel et al. (2020b) and Hewitt et al. (2021) in their diagnostic probing analyses. The crux of our analysis relies on the fact that the encoding extracted by diagnostic probes is not necessarily the functional encoding used by our probed model. Nevertheless, for a model to use a property in its predictions, this property should 2 We focus on grammatical number here. There is, however, also a vast literature investigating how BERT encodes number from a numeracy point of view (Wallace et al., 2019; Geva et al., 2020; Spithourakis and Riedel, 2018). at least be extractable, which is true due to the data processing inequality. In other words, extractability is a necessary, but not sufficient, condition for a property to be used by the model. We quantify the amount of extractable information in a set of representations in terms of a diagnostic probe's V -information (Xu et al., 2020), where the V -information is a direct measure of the amount of extractable information in a random variable. We compute the V -information as: 3 IV ( R  N ) = HV ( N )  HV ( N | R ) (1) where R and N are, respectively, a representation-valued and a number-valued random variables, V is a variation family determined by our diagnostic probe, and the V -entropies are defined as: HV ( N ) = inf q V E n  p ( n ) log 1 q ( n ) (2) HV ( N | R ) = inf q V E n, r  p ( n, r ) log 1 q ( n | r ) (3) Further, if we denote our analyzed model's (i.e., BERT's) hidden representations as: r t,l = model( sentence ) t,l (4) we define our linear diagnostic probe as: p  ( n t = SING | sentence ) =  (   r t,l + b ) (5) where r t,l  R 768 , t is a sentence position and l is a layer, n t is the binary number label associated with the word at position t ,  is the sigmoid function,  is a real-valued column parameter vector and b is a bias term. In this case, we can define our variational family as V = { p  |   R 768 } . We now discuss how we perform a causal intervention to prevent the analyzed model from using a given encoding. The goal is to damage the model and make it forget a property's information. This allows us to analyze whether that encoding actually influences the probed model's predictions i.e., whether this encoding is indeed functional. To this end, we employ amnesic probing (Elazar et al., 2021). 4 In short, we first learn a linear diagnostic classifier, following eq. (5). We then compute 3 See App. B for a detailed description of V -information. 4 In particular, this intervention consists in applying iterative null-space projection to the representations, originally proposed by Ravfogel et al. (2020). We note that Ravfogel et al. (2022a,b) recently proposed two new methods to remove information from a set of representations. By iterating this process, we store a set of parameter vectors  ( k ) and their associated projectors P ( k ) null until we are unable to extract the property. The composition of these projectors makes it possible to remove all linearly extractable number information from the analyzed representations. We can then apply the resulting composition to the said representations to get a new set of vectors: r  t,l = P ( k ) null    P (2)null P (1)null r t,l (7) After learning the projectors, we can measure how erasing a layer's encoding impacts: (i) the subsequent layers, and (ii) our model's performance on the number agreement task. Removing a functional encoding of grammatical number should cause a performance drop on the number agreement task. Further, looking at both (i) and (ii) allows us to make a connection between the amount of information we can extract from our probed model's layers and its behavior. We are thus able to determine whether the encodings revealed by our diagnostic probes are valid from a usage-based perspectiveare they actually used by the probed model on a task that requires them? 5 6 Experimental Setup Data. We perform our analysis on Linzen et al.'s (2016) number agreement dataset, which consists in sentences extracted from Wikipedia. In this dataset, each sentence has been labeled with the position of the cue and target, along with their grammatical number. We assume here that this dataset is representative of the number agreement task; this may not be true in general, however. Model. In our experiments, we probe BERT (De-vlin et al., 2019). 6 Specifically, BERT is a bidirectional transformer model with 12 layers, trained using a masked language modeling objective. As BERT has been shown to perform well on this dataset (Goldberg, 2019), we already know that our probed model passes our first requirement; BERT does use number information in its predictions. 5 Our method differs from amnesic probing mostly in that all our analyses are based on a behavioral task which we know a priori to require the property we investigate. 6 We focus on bert-base-uncased , as implemented in the transformers library (Wolf et al., 2020). Distinguishing Nouns and Verbs. While number is a morpho-syntactic property common to nouns and verbs, we do not know a priori if BERT relies on a single subspace to encode number in their representations. Though it is possible for BERT to use the same encoding, it is equally plausible that each part of speech would get its own number encoding. This leads us to perform our analyses using independent sets of representations for nouns and verbs; as well as a mixed set which merges both of them. Further, verbs are masked when performing the number agreement task, so their representations differ from those of unmasked verbs. Ergo, we analyze both unmasked, and masked tokens at the target verb's positionwhich for simplicity we call verbs and masked verbs, respectively. This leaves us with four probed categories: nouns, verbs, masked verbs, and mixed. In our experiments, we focus on answering two questions: (i) How is number information encoded in BERT's representations? and (ii) How is number information transferred from a noun to its head verb for the model to use it on the behavioral task? We answer question (i) under both extractability and usage-based perspectives. In 7.1, we present our sanity-check experiments that demonstrate that grammatical number is indeed linearly extractable from BERT's representations. In 7.2 and 7.3, we use our causal interventions: we identify BERT's functional encodings of number; and analyze whether these functional encodings are shared across parts of speech. Finally, in 7.4 and 7.5 we investigate question (ii), taking a closer look at the layers in which information is passed. Fig. 1 presents diagnostic probing results in all four of our analyzed settings. 7 A priori , we expect that verbs' and nouns' representations should already contain a large amount of V -information about their grammatical number at the type-level. As expected, we see that the V -information is near its maximum for both verbs and nouns in all layers; this means that nearly 100% of the uncertainty about grammatical number is eliminated given BERT's representations. Further, the mixed category results also reach a maximal V -information, which indicates that it is possible to extract information linearly about both categories at the same time. On the other hand, the V -information of masked verbs is 0 at the non-contextual layer and it progressively grows as we get to the upper layers. 8 As we go to BERT's deeper layers, the V -information steadily rises, with nearly all of the original uncertainty eliminated in the mid layers. This suggests that masked verbs' representations acquire number information in the first 7 layers. However, from these results alone we cannot confirm whether the encoding that nouns and verbs use for number is shared or disjoint. We thus inspect the encoding found by our diagnostic probes, evaluating the cosine similarity between their learned parameters  (ignoring the probes' bias terms b here). If there is a single shared encoding across categories, these cosine similarities should be high. If not, they should be roughly zero. Fig. 2 (left) shows that nouns and verbs might encode number along different directions. Specifically, noun 7 We further present accuracy results in App. A. 8 We note that, in Fig. 1, layer 0 corresponds to the noncontextual representations (i.e. the word embeddings before being summed to BERT's position embeddings). Noncontextual layers thus contain no information about the number of a masked verb, as the mask token contains no information about its replaced verb's number. representations on the first 6 layers seem to have a rather opposite encoding from verbs, while the later layers are mostly orthogonal. Further, while masked verbs and verbs do not seem to share an encoding in the first few layers, they are strongly aligned from layer 6 on (Fig. 2; center). We now know that there are encodings from which we can extract number from nouns and verbs, and that these encodings are disjoint. However, we still do not know whether the encoding is spurious or functional. The patterns previously observed suggest there is a linear encoding, from which grammatical number can be extracted from BERT's representations. We, however, cannot determine whether these encodings are actually those used by the model to make predictions. We now answer this question taking our proposed usage-based perspective, studying the impact of linearly removing number information at both the cue and target positions. 9 We evaluate the model's change in behavior, as evaluated by its performance on the number agreement (NA) task. Fig. 3a and Fig. 3c show the decrease in how much information is extractable at the target position after the interventions are applied. Fig. 3b and Fig. 3d show BERT's accuracy drops on the NA task (as measured at the output level). By comparing these results, we find a strong alignment between the information lost across layers and the damage caused to the performance on the task irreversible information losses resulting from our intervention are mirrored by a performance decrease on the NA task. This alignment confirms that the model indeed uses the linear information erased by our probes. In other words, we have found the probed property's functional encoding. We now return to the question of whether nouns and verbs share a functional encoding of number, or whether BERT encodes number differently for them. To answer this question, we investigate the impact of removing a category's encoding from another category, e.g. applying an amnesic projector learned on verbs to a noun. In particular, we measure how these interventions decrease BERT's performance in our behavioral task. Figs. 3b and 3d presents these results. We observe that each category's projector has a different impact on performance depending on whether it is applied to the cue or the target. Fig. 3b, for instance, shows that using the verb's, or masked verb's, projector to erase information at the cue's (i.e., the noun's) position does not hurt the model. It is similarly unimpactful (as shown in Fig. 3d) to use the noun's projectors to erase a target's (i.e., the masked verb's) number information. Further, the projector learned on the mixed set of representations does affect the cue, but has little effect on the target. Together, these results confirm that BERT relies on rather distinct encodings of number information for nouns and verbs. 10 10 A potential criticism of amnesic probing is that it may remove more information than necessary. Cross-testing our amnesic probes, however, results in little effect on BERT's behavior. It is thus likely that they are not overly harming our model. Further, we also run a control experiment proposed by Elazar et al., removing random directions at each layer (instead of the ones found by our amnesic probes). These results are displayed in the appendix in Tab. 1. These experiments allow us to make stronger claims about BERT's encoding of number information. First, the fact that our interventions have a direct impact on BERT's behavioral output confirms that the encoding we erase actually bears number information as used by the model when making predictions . Second, the observation from Fig. 2that number information could be encoded orthogonally for nouns and verbsis confirmed from a usage-based perspective. Indeed, using amnesic probes trained on nouns has no impact when applied to masked verbs, and amnesic probes trained on verbs have no impact when applied to nouns. These fine-grained differences in encoding may affect larger-scale probing studies if one's goal is to understand the inner functioning of a model. Together, these results invite us to employ diagnostic probes more carefully, as the encoding found may not be actually used by the model. Once we have found which encoding the model uses, we can pinpoint at which layers the information is passed from the cue to the target. To that end, we observe how interventions applied in each layer affect performance. We know number information must be passed from the cue to the target's representationsotherwise the model cannot solve the task. Therefore, applying causal interventions to remove number information should harm the model's behavioral performance when applied to: (i) the cue's representations before the trans-8824 fer occurs; Interestingly, we observe that target interventions are only harmful after the 9 th layer; while noun interventions only hurt up to the 8 th layer (again, shown in Fig. 3). This suggests that the cue passes its number information in the first 8 layers, and that the target stops acquiring number information in the last three layers. While we see a clear stop in the transfer of information after layer 8, Fig. 3a shows that the previous layers' contribution decreases slowly up to that layer. We thus conclude that information is passed in the layers before layer 8; however, we concede that our analysis alone makes it difficult to pinpoint exactly which layers. Finally, in our last experiments, we complement our analysis by performing attention removal to investigate how and where information is transmitted from the cue to the target position. This causal intervention first serves the purpose of identifying the layers where information is transmitted. Further, we wish to understand whether information is passed directly, or through intermediary tokens. To this end, we look at the effect on NA performance after: (i) cutting direct attention from the target to the cue at specific layers, (ii) cutting attention from all tokens to the cue (as information could be first passed to intermediate tokens, which the target could attend to in subsequent layers). 11 Specifically, we perform these interventions in ranges of layers (from layer i up to j ). We report number agreement accuracy drops in Fig. 4. 12 The diagonals from this figure show that removing attention from a single layer has basically no effect. Further, cutting attention from layers 6 to 10 suffices to observe near-maximal effect for direct attention. Interestingly, it is at those layers where we see a transition from it being more harmful to apply amnesic projectors to the cue or to the target (in 7.4). However, while those layers play a role in carrying number information to the target position, the drop is relatively modest when cutting only direct attention (  10% ). Cutting attention from all tokens to the cue, in turn, has a significant effect on performance (up to  40% ), and is maxi-11 Klafka and Ettinger (2020), for instance, showed that number information of a given token was distributed to neighboring tokens in the upper layers 12 We detail these interventions in App. mal for layers 2 to 8. This first suggests that, while other clues in the sentence could indicate the target verb's number (such as a noun's determiner), the noun itself is the core source of number information. Further, this shows the target can get information from intermediate tokens, instead of number being passed exclusively through direct attention. 13 8 Conclusion Our analysis of grammatical number allows us to track how a simple morpho-syntactic property, grammatical number, is encoded across BERT's layers and where it is transferred between them before being used on the model's predictions. Using carefully chosen causal interventions, we demonstrate that forgetting number information impacts both: (i) BERT's behavior and (ii) how much information is extractable from BERT's inner layers. Further, the effects of our interventions on these two, i.e., behavior and information extractability, line up satisfyingly, and reveal the encoding of number to be orthogonal for nouns and verbs. Finally, we are also able to identify the layers in which the transfer of information occurs, and find that the information is not passed directly but through intermediate tokens. Our ability to concretely evaluate our interventions' impact is due to our focus on grammatical number and the number agreement taskwhich directly align probed information and behavioral performance. The authors foresee no ethical concerns with the work presented in this paper. We thank Josef Valvoda, the anonymous reviewers, and the meta-reviewer, for their invaluable feedback in improving this paper. Karim Lasri's work is funded by the French government under management of Agence Nationale de la Recherche as part of the Investissements d'avenir program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Insti-tute). Ryan Cotterell acknowledges support from the Swiss National Science Foundation (SNSF) as part of the The Forgotten Role of Inductive Bias in Interpretability project. Tiago Pimentel is supported by a Meta Research PhD Fellowship.
abstain abstain result objective method method objective result result method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method result method abstain abstain abstain method abstain abstain abstain abstain objective objective abstain objective result method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other other method other other other other objective method abstain method abstain method abstain abstain abstain method method abstain abstain abstain abstain method method method abstain method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method other other other other