Our goal is procedural text comprehension, namely tracking how the properties of entities (e.g., their location) change with time given a procedural text (e.g., a paragraph about photosynthesis, a recipe). This task is challenging as the world is changing throughout the text, and despite recent advances, current systems still struggle with this task. Our approach is to leverage the fact that, for many procedural texts, multiple independent descriptions are readily available, and that predictions from them should be consistent (label consistency). We present a new learning framework that leverages label consistency during training, allowing consistency bias to be built into the model. Evaluation on a standard benchmark dataset for procedural text, ProPara (Dalvi et al., 2018), shows that our approach significantly improves prediction performance (F1) over prior state-of-the-art systems. We address the task of procedural text comprehension, namely tracking how the properties of entities (e.g., their location) change with time throughout the procedure (e.g., photosynthesis, a cooking recipe). This ability is an important part of text understanding, allowing the reader to infer unstated facts such as how ingredients change during a recipe, what the inputs and outputs of a scientific process are, or who met whom in a news article about a political meeting. Although several procedural text comprehension systems have emerged recently (e.g., EntNet (Hena  et al., 2017), NPN (Bosselut et al., 2018), and ProStruct (Tandon et al., 2018)), they still make numerous prediction errors. A major challenge is that fully annotated training data for this task is expensive to collect, because * Work done while at the Allen Institute for Artificial Intelligence. To address this challenge, and thus improve performance, our goals are two-fold: first, to better leverage the training data for procedural text comprehension that is available, and second, to utilize additional unlabeled data for the task (semi-supervised learning). Our approach in each case is to exploit label consistency , the property that two distinct texts covering the same procedure should be generally consistent in terms of the state changes that they describe, which constitute the labels to be predicted for the text. For example, in di  erent texts describing photosynthesis, we expect them to be generally consistent about what happens to oxygen (e.g., that it is created), even if the wordings di  er (Figure 1). Using multiple, distinct passages to understand a process or procedure is challenging. Although the texts describe the same process, they might express the underlying facts at di  erent levels of granularity, using di  erent wordings, and including or omitting di  erent details. As a result, the details may di  er between paragraphs, making them hard to align and to check for consistency. Nonetheless, even if the details di  er, we conjecture that the top-level summaries of each paragraph, which Figure 2: Three (simplified) passages from ProPara describing photosynthesis, the (gold) state changes each entity undergoes at each step s 1 , s 2 , ..., s T , and the summary of state changes that each entity undergoes (an aggregation of the step-by-step changes), where M = MOVED, D = DESTROYED, C = CREATED . describe the types of state change that each entity undergoes, will be mostly consistent. For example, although independent texts describing photosynthesis vary tremendously, we expect them to be consistent about what generally happens to sugar, e.g., that it is created (Figure 2). In this paper, we introduce a new training framework, called L a CE (Label Consistency Explorer), that leverages label consistency among paragraph summaries. In particular, it encourages label consistency during end-to-end training of a neural model, allowing consistency bias to improve the model itself, rather than be enforced in a post-processing step, e.g., posterior regularization (Ganchev et al., 2010). We evaluate on a standard benchmark for procedural text comprehension, called ProPara (Dalvi et al., 2018). We show that this approach achieves a new state-of-the-art performance in the fully supervised setting (when all paragraphs are annotated), and also demonstrate that it improves performance in the semi-supervised setting (us-ing additional, unlabeled paragraphs) with limited training data. In the latter case, summary predictions from labeled data act as noisy gold labels for the unlabeled data, allowing additional learning to occur. Our contributions are thus: 1. A new learning framework, L a CE, applied to procedural text comprehension that improves the label consistency among di  erent paragraphs on the same topic. 2. Experimental results demonstrating that L a CE achieves state-of-the-art performance on a standard benchmark dataset, ProPara, for procedural text. Leveraging Label Consistency Leveraging information about label consistency (i.e., similar instances should have consistent labels at a certain granularity) is an e  ective idea. It has been studied in computer vision (Haeusser et al., 2017; Chen et al., 2018) and IR (Clarke et al., 2001; Dumais et al., 2002). Learning by association (Haeusser et al., 2017) establishes implicit cross-modal links between similar descriptions and leverage more unlabeled data during training. Schtze et al. (2018); Hangya et al. (2018) adapt the similar idea to exploit unlabeled data for the cross-lingual classifica-tion. We extend this line of research in two ways: by developing a framework allowing it to be applied to the task of structure prediction; and by incorporating label consistency into the model itself via end-to-end training, rather than enforcing consistency as a post-processing step. Semi-supervised Learning Approaches Besides utilizing the label consistency knowledge, our learning framework is also able to use unlabeled paragraphs, which fits in the literature of semi-supervised learning approaches (for NLP). Zhou et al. (2003) propose an iterative label propagation algorithm similar to spectral clustering. Zhu et al. (2003) propose a semi-supervised learning framework via harmonic energy minimization for data graph. Talukdar et al. (2008) propose a graph-based semi-supervised label propagation algorithm for acquiring open-domain labeled classes and their instances from a combination of unstructured and structured text sources. Our framework extends these ideas by introducing the notion of groups (examples that are expected to be similar) and summaries (what similarities are expected), applied in an end-to-end-framework. Procedural Text Understanding and Reading Comprehension There has been a growing interest in procedural text understanding / QA recently. The ProcessBank dataset (Berant et al., 2014) asks questions about event ordering and event arguments for biology processes. bAbI (Weston et al., 2015) includes questions about movement of entities, however it's synthetically generated and with a small lexicon. Kiddon et al. (2015)'s RECIPES dataset introduces the task of predicting the locations of cooking ingredients, and Kiddon et al. (2016) for recipe generation. In this paper, we continue this line of exploration using ProPara, and illustrate how the previous two lines of work (label consistency and semi-supervised learning) can be integrated. A general condition for applying our method is having multiple examples where, for some properties, we expect to see similar values. For example, for procedural text, we expect paragraphs about the same process to be similar in terms of which entities move, are created, and destroyed; for different news stories about a political meeting, we expect top-level features (e.g., where the meeting took place, who attended) to be similar; for di  erent recipes for the same item, we expect loosely similar ingredients and steps; and for di  erent images of the same person, we expect some high-level characteristics (e.g., height, face shape) to be similar. Note that this condition does not apply to every learning situation; it only applies when training examples can be grouped, where all group members are expected to share some characteristics that we can identify (besides the label used to form the groups in the first place). More formally, for training, the input is a set of labeled examples ( x gi , y gi ) (where y gi are the labels for x gi ), partitioned into G groups, where the g subscript denotes which group each example belongs to. Groups are defined such that examples of the same group g are expected to have similar labels for a subset of labels y gi . We call this subset the summary labels . We assume that both the groupings and the identity of the summary labels are provided. The output of training is a model M for labeling new examples. For testing, the input is the model M and a set of unlabeled (and ungrouped) examples x t , and the output are their predicted labels  y t . Note that this formulation is agnostic to the learning algorithm used. Later, we will consider both the fully supervised setting (all training examples are labeled) and semi-supervised setting (only a subset are labeled). We instantiate this framework for procedural text comprehension, using the ProPara task (Dalvi et al., 2018). In this task, x gi are paragraphs of text describing a process (e.g., photosynthesis), the labels y gi describe the state changes that each entity in the paragraph undergoes at each step (sentence) (e.g., that oxygen is created in step 2), and the groups are paragraphs about the same topic (ProPara tags each paragraph with a topic, e.g., there are three paragraphs in ProPara describing photosynthesis). More precisely, each x gi consists of:  the name (topic) of a process, e.g., photosynthesis  a sequence (paragraph) of sentences S = [ s 1 , ..., s T ] that describes that process  the set of entities E mentioned in that text, e.g., oxygen, sugar and the targets (labels) to predict are: Figure 3: Example of batches constructed from a group (here, the group contains three labeled examples x 1 , x 2 , x 3 ). the state changes that each entity in E undergoes at each step (sentence) of the process, where a state change is one of { Moved,Created,Destroyed,None }. These state changes can be conveniently expressed using a | S |  | E | matrix (Figure 2). State changes also include arguments, e.g., the source and destination of a move. We omit these in this paper to simplify the description. Finally, we define the summary labels as the set of state changes that each entity undergoes at some point in the process, without concern for when. For example, in Passage 1 in Figure 2, CO 2 is Moved (M) and Destroyed (D), while sugar is Created (C). These summary labels can be computed from the state-change matrix by aggregating the state changes for each entity over all steps. Our assumption here is that these summaries will generally be the same (i.e., consistent) for di  erent paragraphs about the same topic. L a CE then exploits this assumption by encouraging this inter-paragraph consistency during training, as we now describe. While a traditional supervised learning model operates on individual examples, L a CE operates on batches of grouped examples X g . Given a group g containing N labeled examples { x 1 , ..., x N } (we drop the g subscript for clarity), L a CE creates N batches, each containing all the examples but with a di  erent x i labeled as primary, along with the gold labels y i for (only) the primary example. (We informally refer to the primary example as the first example in each batch). Then for each batch, L a CE jointly optimizes the usual supervised loss L sup (  y i , y i ) for the primary example, along with a consistency loss between (summary) predictions for all other members of the group and the primary example, L con (  y j ,  y i ) for all j (cid:44) i . This is illustrated in Figures 4 and 3. This is repeated for all batches. For example, for the three paragraphs about photosynthesis (Figure 2), batch 1 compares the first paragraph's predictions with its gold labels, and also compares the summary predictions of paragraphs 2 and 3 with those of the first paragraph (Figure 3). This is then repeated using paragraph 2, then paragraph 3 as primary. The result is that L a CE jointly optimizes the supervised loss L sup and consistency loss L con to train a model that is both accurate for the given task as well as consistent in its predictions across examples that belong to the same group. This process is approximately equivalent to jointly optimizing the usual supervised loss L sup (  y i , y i ) for all examples in the group, and the pairwise consistency loss L con (  y j ,  y i ) for all pairs ( x j , x i ) , j (cid:44) i in the group. However, there is an important di  erence, namely the relative contributions of L sup and L con is varied among batches, depending on how accurate the predictions for the primary example are (i.e., how small L sup is), as we describe later in Section 4.3. This has the e  ect of paying more attention to consistency loss when predictions on the primary are more accurate. We also extend L a CE to the semi-supervised setting as follows. For the semi-supervised setting, where only m of n ( m < n ) examples are labeled, we only form m batches, where each batch has  " M o d e l  #  $ Predicted state changes Gold state changes Label loss:  '() back-propagate combined loss A batch for group  + , Consistency loss:  -./  " We now describe how L a CE is applied to our goal of comprehending procedural text. Note that L a CE is agnostic to the learner used within the framework. For this application, we use a simpli-fied version of ProStruct (Tandon et al., 2018), a publicly available system designed for the ProPara task. Our implementation simplifies ProStruct by reusing its encoder, but then predicting (a distribution over) each state change label independently during decoding for every cell in the | S |  | E | grid (Figure 2). We briefly summarize this here. ProStruct uses an encoder-decoder architecture that takes procedural text as input and predicts the state changes of entities E in the text as output. During encoding, each step s t is encoded using | E | embeddings, one for each entity e j  E . Each embedding represents the action that s t describes, applied to e k . The model thus allows the same action to have di  erent e  ects on di  erent entities (e.g., a transformation destroys one entity, and creates another). For each ( s t , e j )  S  E pair, the step is fed into a BiLSTM (Hochreiter and Schmidhuber, 1997), using pretrained GloVe (Pennington et al., 2014) vectors v w for each word w i concatenated with two indicator variables, one indicating whether w i is a word referring to e j , and one indicating whether w i is a verb. A bilinear attention layer then computes attention over the contextualized vectors h i output by the BiLSTM: a i = h i  B  h ev + b , where B and b are learned parameters, and h ev is the concatenation of h e (the averaged contextualized embedding for the entity words w e ) and h v (the averaged contextualized embedding for the verb words w v ). Finally, the output vector c tj is the attention-weighted sum of the h i : c tj = (cid:80) Ii = 1 a i  h i . Here, c tj can be thought of as representing the action s t applied to entity e j . This is repeated for all steps and entities. To decode the action vectors c tj into their resulting state changes they imply, each is passed through a feedforward layer to generate logit (  tj ), a set of logistic activations over the K possible state changes  tj for entity e j in step s t . For ProPara, there are K = 4 possible state changes: Move, Create, Destroy, and None . These logits form a distribution over possible state changes to predict, for each entity and step in the text. We then compute loss, described next, using these distributions directly rather than discretizing them into exact predictions at this stage, so as not to lose information. We start by creating training batches for each X g i  X g . From a group X g i comprising of n examples, we create n training batches. A batch consists of all n examples ( x 1 , x 2 , ..., x n ), but the loss computation is di  erent in each batch. Figure 3 illustrates this. The loss computation in a batch is based on the usual supervised loss and additionally the consistency loss, as follows: Here, L sup (  y 1 , y 1 ) is the negative log likelihood loss * against the gold labels y 1 , and  is a hyperpa-rameter tuned on the dev set. To compute the consistency loss L con (  y i ,  y 1 ), we compare the summaries computed from  y i and  y 1 . In our particular application, a summary lists all the state changes each entity undergoes, formed by aggregating its step-by-step state changes. For example, for paragraph x 1 in Figure 4, as CO 2 first moves ( M ), then later is destroyed ( D ), we summarize its state changes as s ( CO 2 ,  y 1 ) = { M,D }. In practice, as our decoder outputs distributions over the four possible values { M,C,D,N } rather than a single value, we summarize by adding and normalizing these distributions, producing a summary distribution s ( e ,  y j ) over the four values rather than a discrete set of values. To compute the consistency loss L con (  y i ,  y 1 ) itself, we compare summaries for each entity e that occurs in both paragraph x 1 and paragraph x i (re-ferred to as Ent( x 1 ) and Ent( x i ) respectively), and compute the average mean squared error (MSE) between their summary distributions. We also tried other alternatives (e.g., Kullback-Leibler divergence) for calculating the distance between summary distributions, but mean squared error per* Loss function L sup is exactly same as the loss function used in the base model so that we can measure the e  ect of adding consistency loss. forms best. Equation 2 shows the details for computing the consistency loss. Note that each paragraph contains varying number of entities and sentences. It is possible that some paragraphs do not mention exactly the same entities as the labeled paragraph (first element in the batch). In such cases, we penalize the model only for predictions for co-occurring entities. Unmatched entities are not penalized. The supervised loss L sup (  y 1 , y 1 ) is large in the early epochs when the model is not su  ciently trained. At this point, it is beneficial for the model to pay no attention to the consistency loss L con (  y j ,  y 1 ) as the predicted action distributions are inaccurate. To implement this, if L sup is above a defined threshold then the consistency loss term in Equation 1 is ignored (i.e.  = 1). Otherwise, Equation 1 is used as is. This can loosely be seen as a form of simulated annealing (Kirkpatrick et al., 1988), using just two temperatures. Note that the time (epoch number) when the temperature (lambda) changes will vary across batches depending on the supervised loss within that batch of data, hence we call it an adaptive loss. We now present results on ProPara, the procedural text comprehension dataset introduced in (Dalvi et al., 2018). There are 187 topics in this dataset and a total of 488 labeled paragraphs (around 3 labeled paragraphs per topic). The task is to track how entities change state through the paragraph (as described in Section 3.2) and answer 4 classes of questions about those changes (7043 / 913 / 1095 questions in each of the train / dev / test partitions re-spectively). We compare L a CE with the baselines and prior state-of-the-art model ProStruct (Tandon et al., 2018) in two settings: (1) Fully supervised learning (using all the training data). (2) Semi-supervised learning (using some or all of the training data, plus additional unlabeled data). We evaluated L a CE by comparing its performance against published, state-of-the-art results on ProPara, using the full training set to train L a CE. The results are shown in Table 1. In Table 1, all the baseline numbers are the results reported in (Tan-don et al., 2018). Note that all these baselines are trying to reduce the gap between predicted labels and gold labels on the training dataset. L a CE, however, also optimizes for consistency across labels for groups of paragraphs belonging to the same topic. As L a CE uses parts of ProStruct as its learning algorithm, the gains over ProStruct appear to be coming directly from its novel learning framework described in Section 4.1. To confirm this, we also performed an ablation study, removing the consistency loss term and just using the base model in L a CE. The results are shown in Table 2, and show that the F1 score drops from 56.6 to 53.2, illustrating that the consistency loss is responsible for the improvement. In addition, Table 2 indicates that consistency loss helps improve both precision and recall. Also note that L a CE simplifies parts of ProStruct. For example, unlike ProStruct, L a CE does not use a pre-computed knowledge base during decoding. Thus L a CE is more e  cient to train than ProStruct ( > 15x faster at training time). Unlike the other systems in Table 1, L a CE is able to use unlabeled data during training. As described in Section 4.1, given a group containing both labeled and unlabeled paragraphs, we create as many batches as the number of labeled paragraphs in the group. Hence, paragraphs x i with gold labels y i can contribute to both supervised loss L sup and consistency loss L con . Additionally, we can use unlabeled Models Proportion of labeled paragraphs used per training topic 33% 66% 100% P ro S truct 45.4 50.6 54.5 L a CE 47.3 51.2 56.6 L a CE + unlabeled data 49.9 52.9 56.7 Table 3: Comparing L a CE vs. P ro S truct with varying amount of labeled paragraphs available per training topic. paragraphs x j (i.e., without gold labels y j ), while computing consistency loss L con . This way L a CE can make use of unlabeled data during training. To evaluate this, we collected 877 additional unlabeled paragraphs for ProPara topics  . As the original ProPara dataset makes some simplifying assumptions, in particular that events are mentioned in chronological order, we used Mechanical Turk to collect additional paragraphs that conformed to those assumptions (rather than collecting paragraphs from Wikipedia, say). Approximately 3 extra paragraphs were collected for each topic in ProPara. Note that collecting unlabeled paragraphs is substantially less expensive than labeling paragraphs. We then trained the P ro S truct and L a CE models varying two di  erent parameters: (1) the percentage of the labeled (ProPara) training data used to train the system (2) for L a CE only, whether the additional unlabeled data was also used. This allows us to see performance under di  erent conditions of sparsity of labeled data, and (for L a CE) also assess how much unlabeled data can help under those conditions. During training, the unused labeled data was ignored (not used as unlabeled data). We keep the dev and test partitions the same as original dataset, picking a model based on dev performance and report results on test partition. The results are shown in Table 3. In the first two rows, P ro S truct and L a CE are both trained with x% of labeled data, while the last row reports perThe unlabeled paragraphs are available at http://data. allenai.org/propara/ . Table 3 demonstrates that L a CE results in even larger improvements over P ro S truct when the amount of labeled data is limited. In addition, unlabeled data adds an additional boost to this performance, in particular when labeled data is sparse. Further examination suggests that the gains in F1 are resulting mainly from improved recall, as shown in Figure 5. We believe that having access to unlabeled paragraphs and optimizing consistency across paragraphs for training topics, helps L a CE generalize better to unseen topics. We implement our proposed model L a CE in Py-Torch (Paszke et al., 2017) using the AllenNLP (Gardner et al., 2018) toolkit. We added a new data iterator that creates multiple batches per topic (Figure 3) which enables easy computation of consistency loss. We use 100D Glove embeddings (Pennington et al., 2014), trained on Wikipedia 2014 and Gigaword 5 corpora (6B tokens, 400K vocab, uncased). Starting from glove embeddings appended by entity and verb indicators, we use bidirectional LSTM layer to create contextual representation for every word in a sentence. We use 100D hidden representations for the bidirectional LSTM (Hochreiter and Schmidhuber, 1997) shared between all inputs (each direction uses 50D hidden vectors). We use attention layer on top of BiLSTM, using a bilinear similarity function similar to (Chen et al., 2016) to compute attention weights over the contextual embedding for each word in the sentence. To compute the likelihood of all state changes Consistency Score (%) Train Test ProStruct 46.70 37.21 L a CE 54.39 38.36 Table 5: Consistency score comparison individually, we use a single layer feedforward network with input dimension of 100 and output 4. In these experiments, we check if the supervised loss L sup is less than a threshold (0.2 in our case) then we use equation 1 and lambda = 0 . 05. All hyper-parameters are tuned on the dev data. During training we use multiple paragraphs for a topic to optimize for both supervised and consistency loss. At test time, L a CE's predictions are based on only one given paragraph. All the performance gains are due to the base model being more robust due to proposed training procedure. The code for L a CE model is published at https://github.com/allenai/propara . We first discuss the predicted label consistency across paragraphs for L a CE vs. P ro S truct . We then identify some of the limitations of L a CE. L a CE attempts to encourage consistency between paragraphs about the same topic during training, and yield similar benefit at test time. To examine whether this happens in practice, we compute and report the consistency score between paragraphs about the same topic (Table 5). Specifically, for an entity that appears in two paragraphs about the same topic, we compare whether the summaries of state change predictions for each match. The results are shown in Table 5. The table shows that L a CE achieves greater prediction consistency during training, and that this benefit plays out to some extent at test time even though label consistency is not enforced at test time (we do not assume that examples are grouped at test time, hence consistency between groups cannot be enforced as the grouping is unknown). As an illustration, for the topic describe the life cycle of a tree which is unseen at training time, for the three paragraphs on the topic, ProStruct predicts that tree is created; not-changed; and created respectively, while L a CE correctly predicts that tree is created; created; and created respectively. To understand L a CE's behavior further, we examined cases where L a CE's and P ro S truct 's predictions di  er, and examined their agreement with gold labels. In this analysis we found three major sources of errors for L a CE:  The label consistency assumption does not always hold: In Section 3.1, we explain that L a CE relies on summary labels being consistent across examples in the same group. We found that for some of the topics in our training dataset this assumption is sometimes violated. E.g., for the topic How does the body control its blood sugar level? , there are two di  erent paragraphs; one of them describes the entity sugar as being Created and then Destroyed to create bloodsugar , while the other paragraph describes the same event in a di  erent way by saying that the entity sugar is Created and then Moved to the blood. L a CE can thus goes wrong when trying to enforce consistency in such cases. Lexical variance between entities across paragraphs: Di  erent paragraphs about the same topic may describe the procedure using di  erent wordings, resulting in errors. For example, in paragraphs about the topic what happens during photosynthesis? , the same entity ( carbon dioxide ) is referred to by two di  erent strings, CO 2 in one paragraph and carbon dioxide in another. Currently, L a CE does not take into account entity synonyms, so it is unable to encourage consistency here. An interesting line of future work would be to use the embedding space similarity between entity names, to help address this problem. improve consistency: For the topic Describe how to make a cake at training time, when presented with two paragraphs, L a CE tries to be consistent and incorrectly predicts that cake is Destroyed in both paragraphs. ProStruct does not attempt to improve prediction consistency, here resulting in less consistent but in this case more accurate predictions for this topic. 5.5 Directions For Enhancing L a CE  Improve L a CE for ProPara: L a CE's performance on ProPara can be improved further by a) soft matching of entities across paragraphs instead of current exact string match b) exploring more systematic ways (e.g., simulated annealing) to define adaptive loss c) using additional sources of unlabeled data (e.g., web, textbooks) weighed by their reliability. Apply L a CE on other tasks: Architecturally, L a CE is a way to train any existing structured prediction model for a given task to produce consistent labels across similar data-points. Hence it can be easily applied to other tasks where parallel data is available (group-ing function) and there is a way to e  ciently compare predictions (summary labels) across parallel datapoints, e.g. event extraction from parallel news articles (Chinchor, 2002). Further, summary labels need not be action categories (e.g., Created , Destroyed ). Consistency can also be computed for QA task where multiple parallel text is available for reading comprehension. We plan to explore this direction in the future. Our goal is procedural text comprehension, a task that current systems still struggle with. Our approach has been to exploit the fact that, for many procedures, multiple independent descriptions exist, and that we expect some consistency between those descriptions. To do this, we have presented a taskand model-general learning framework, L a CE, that can leverage this expectation, allowing consistency bias to be built into the learned model. Applying this framework to procedural text, the resulting system obtains new state-of-the-art results on the ProPara dataset, an existing benchmark for procedural text comprehension. It also demonstrates the ability to benefit from unlabeled paragraphs (semi-supervised learning), something that prior systems for this task were unable to do. We have also identified several avenues for further improvement (Section 5.4), and are optimistic that further gains can be achieved. Computations on beaker.org were supported in part by credits from Google Cloud.
objective abstain abstain objective result method abstain abstain abstain objective method method abstain abstain abstain method abstain method objective abstain method objective abstain objective objective other other other other objective method other other other objective other other other other method abstain method abstain abstain abstain abstain method abstain abstain abstain method method other other abstain abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain result other