We present an approach to minimally supervised relation extraction that combines the benefits of learned representations and structured learning, and accurately predicts sentence-level relation mentions given only proposition-level supervision from a KB. By explicitly reasoning about missing data during learning, our approach enables large-scale training of 1D convolutional neural networks while mitigating the issue of label noise inherent in distant supervision. Our approach achieves state-of-the-art results on minimally supervised sentential relation extraction, outperforming a number of baselines, including a competitive approach that uses the attention layer of a purely neural model. 1 1 Introduction Recent years have seen significant progress on tasks such as object detection, automatic speech recognition and machine translation. These performance advances are largely driven by the application of neural network methods on large, high-quality datasets. In contrast, traditional datasets for relation extraction are based on expensive and time-consuming human annotation (Doddington et al., 2004) and are therefore relatively small. Distant supervision (Mintz et al., 2009), a technique which uses existing knowledge bases such as Freebase or Wikipedia as a source of weak supervision, enables learning from large quantities of unlabeled text and is a promising approach for scaling up. Recent work has shown promising results from large-scale training of neural networks for relation extraction (Toutanova et al., 2015; Zeng et al., 2015). There are, however, significant challenges due to the inherent noise in distant supervision. For 1 Our code and data are publicly available on Github: https://github.com/bflashcp3f/PCNN-NMAR example, Riedel et al. (2010) showed that, when learning using distant supervision from a knowledge base, the portion of mis-labeled examples can vary from 13% to 31%. To address this issue, another line of work has explored structured learning methods that introduce latent variables. An example is MultiR (Hoffmann et al., 2011), which is based on a joint model of relations between entities in a knowledge base and those mentioned in text. This structured learning approach has a number of advantages; for example, by integrating inference into the learning procedure it has the potential to overcome the challenge of missing facts by ignoring the knowledge base when mention-level classifiers have high confidence (Ritter et al., 2013; Xu et al., 2013). Prior work on structured learning from minimal supervision has leveraged sparse feature representations, however, and has therefore not benefited from learned representations, which have recently achieved state-of-the-art results on a broad range of NLP tasks. In this paper, we present an approach that combines the benefits of structured and neural methods for minimally supervised relation extraction. Our proposed model learns sentence representations that are computed by a 1D convolutional neural network (Collobert et al., 2011) and are used to define potentials over latent relation mention variables. These mention-level variables are related to observed facts in a KB using a set of deterministic factors, followed by pairwise potentials that encourage agreement between extracted propositions and observed facts, but also enable inference to override these soft constraints during learning, allowing for the possibility of missing information. Because marginal inference is intractable in this model, a MAP-based approach to learning is applied (Taskar et al., 2004). Our approach is related to recent work structured learning with end-to-end learned representations, including Structured Prediction Energy Networks (SPENs) (Belanger and McCallum, 2016); the key differences are the application to minimally supervised relation extraction and the inclusion of latent variables with deterministic factors, which we demonstrate enables effective learning in the presence of missing data in distant supervision. Our proposed method achieves state-of-the-art results on minimally supervised sentential relation extraction, outperforming a number of baselines including one that leverages the attention layer of a purely neural model (Lin et al., 2016). In this section we present our model, which combines continuous representations with structured learning. We first review the problem setting and introduce notation, next we present our approach to extracting feature representations which is based on the piecewise convolutional neural network (PCNN) model of Zeng et. al. (2015) and includes positional embeddings (Collobert et al., 2011). Finally we describe how this can be combined with structured latent variable models that reason about overlapping relations and missing data during learning. Given a set of sentences, s = s 1 , s 2 . . . , s n that mention a pair of knowledge base entities e 1 and e 2 (dyad), our goal is to predict which relation, r , is mentioned between e 1 and e 2 in the context of each sentence, represented by a set of hidden variables, z = z 1 , z 2 , . . . z n . Relations are selected from a fixed set drawn from a knowledge base, in addition to NA (no relation). Minimally supervised learning is more difficult than supervised relation extraction, because we do not have direct access to relation labels on the training sentences. Instead, during learning, we are only provided with information about what relations hold between e 1 and e 2 according to the KB. The problem is further complicated by the fact that most KBs are highly incomplete (this is the reason we want to extend them by extracting information from text in the first place), which effectively leads to false-negatives during learning. Furthermore, there are many overlapping relations between dyads, so it is easy for a model trained using minimal supervision from a KB to confuse these relationships. All of these issues are addressed to some degree by the structured learning approach that we present in Section 2.3. First, however we present our approach to feature representation based on convolutional neural networks. In the following section we review the Piecewise CNN (PCNN) architecture, first proposed by Zeng et. al. (2015), which is used as the basis for our feature representation. Input Representation: A sentence, s i consisting of l words is represented by two types of embeddings: word embeddings, E i , and position embeddings, P i relative to the entity pair. Following Lin et. al. (2016), word embeddings were initialized by running Word2Vec on the New York Times corpus and later fine-tuned; position embeddings encode the position of the word relative to KB entities, e 1 and e 2 , mentioned in the sentence. The form of input sentence representation is w 1 , w 2 ,    , w l , where w i  R d . The dimension of embedding at each word position is equal to the word embedding dimension plus two times the position embedding size (one position is encoded for each entity). Convolution: Given an input sentence representation, we perform 1D convolution within a window of length l to extract local features. Assume we have d f convolutional filters ( F = { f 1 , f 2 ,    , f d f } , f i  R l  d ) . The output of the i -th convolutional filter within the j -th window is: c ij = f i  w j  l +1: j + b (1  j  m + l  1) Where b is a bias term. We use zero padding when the window slides out of the sentence boundaries. Piecewise Max Pooling: The output of the convolutional layer c i is separated into three parts ( c i 1 , c i 2 , c i 3 ) using the positions of the two entities in the sentence. Max pooling over time is then applied to each of these parts, followed by an el-ementwise tanh. The final sentence vector is defined as follows: [ x ] ik = tanh(max j ( c ikj )) (1  i  d f , 1  k  3) 2.3 Structured Minimally Supervised Learning Our proposed model is based on the PCNN representations described above, in addition to a latent variable model that reasons about missing data and ambiguous relations during learning and is illustrated in Figure 1. The embedding for sentence i , is used to define a factor over the i th input sentence and latent relation mention variable z i :  PCNN ( s i , z i ) = e x i   zi where x i is the representation for sentence s i , as encoded by the piecewise CNN. Another set of factors,  OR , link the sentence-level mention variables, z i , to aggregate-level variables t j , representing whether relation j is mentioned between e 1 and e 2 in text. This is modeled using a deterministic OR:  OR ( z , t j ) = 1  t j  i : j = z i where 1 x is an indicator function that takes the value 1 when x is true. The choice of deterministic OR can be interpreted intuitively as follows: if a proposition is true according to t j , then it must be extracted from at least one sentence in the training corpus, on the other hand, if it is false, no sentences in the corpus can mention it. Finally, we incorporate a set of factors that penalize disagreement between observed relations in the KB, d j , and latent variables t j , which represent whether relation j was extracted from the text. The penalties for disagreement with the KB are hyperparameters that are adjusted on held-out development data and incorporate entity frequency information from the KB, to model the intuition that more popular entities are less likely to have missing facts:  A ( t j , d j ) =   e   T , if t j = 0 and d j = 1 e   D , if t j = 1 and d j = 0 1 , otherwise Putting everything together, the (unnormalized) joint distribution over t , d and z conditioned on sentences s mentioning a dyad is defined as follows: P ( d , t , z | s )  | s | (cid:89) i =1  PCNN ( s i , z i )  (cid:16) | r | (cid:89) j =1  OR ( z , t j )  A ( t j , d j ) (cid:17)  = exp( S  ( s , z , t , d )) (1) Here,  is a tunable hyperparameter to adjust impact of the disagreement penalty, and S  (  ) is the model score for a joint configuration of variables, which corresponds to the log of the unnormalized probability. A standard conditional random field (CRF) formulation would optimize model parameters,  so as to maximize marginal probability of the observed KB relations, d conditioned on observed sentences, s : P ( d | s ) = (cid:88) z , t P ( d , t , z | s ) Computing gradients with respect to P ( d | s ) (and marginalizing out z and t ) is computationally intractable, so instead we propose an approach that uses maximum-a-posteriori (MAP) parameter learning (Taskar et al., 2004) and is inspired by the latent structured SVM (Yu and Joachims, 2009). Given a large text corpus in which a set of sentences, s mention a specific pair of entities ( e 1 , e 2 ) and a set of relations d hold between e 1 and e 2 , our goal is to minimize the structured hinge loss: LSH (  ) = max   0 , max z  e , t  e , d  e [ S  ( s , z  e , t  e , d  e ) + l Ham ( d  e , d )]  max z  g , t  g (cid:2) S  ( s , z  g , t  g , d ) (cid:3)   (2) Where l Ham ( d  e , d ) is the Hamming distance between the bit vector corresponding to the set of observed relations holding between ( e 1 , e 2 ) in the KB and those predicted by the model. Minimizing LSH (  ) can be understood intuitively as adjusting the parameters so that configurations consistent with observed relations in the KB, d , achieve a higher model score than those with a large hamming distance from the observed configuration. z  e corresponds to the most confusing configuration of the sentence-level relation mention variables (i.e. one that has a large score and also a large Hamming loss) and z  g corresponds to the best configuration that is consistent with the observed relations in the KB. This objective can be minimized using stochastic subgradient descent. Fixing z  g and z  e to their maximum values in Equation 2, subgradients with respect to the parameters can be computed as follows:   LSH (  ) =   0 if LSH (  )  0 ,   S  ( s , z  e , t  e , d  e )   S  ( s , z  g , t  g , d ) otherwise (3) =   0 if LSH (  )  0 , (cid:80) i   log  PCNN ( s i ,z  e,i )  (cid:80) i   log  PCNN ( s i ,z  g,i ) otherwise (4) Because the second factor of the product in Equation 1 does not depend on  , it is straightforward to compute subgradients of the scoring function,  S  (  ) , with fixed values of z  g and z  e using backpropagation (Equation 4). Inference: The two inference problems, corresponding to maximizing over hidden variables in Equation 2 can be solved using a variety of solutions; we experimented with A  search over left-to-right assignments of the hidden variables. An admissible heuristic is used to upper-bound the maximum score of each partial hypothesis by maximizing over the unassigned PCNN factors, ignoring inconsistencies. This approach is guaranteed to find an optimal solution, but can be slow and memory intensive for problems with many variables. In preliminary experiments on development data, we found that local-search (Eisner and Tromble, 2006) using both relation type and mention search operators (Liang et al., 2010; Ritter et al., 2013) usually finds an optimal solution and also scales up to large training datasets; we use local search with 30 random restarts to compute argmax assignments for the hidden variables, z  g and z  e , in all our experiments. Bag-Size Adaptive Learning Rate: Since the search space of the MAP inference problem increases exponentially as the number of hidden variables goes up, it becomes more difficult to find the exact argmax solution using local search, leading to increased noise in the computed gradients. To mitigate the search-error problem in large bags of sentences, we dynamically modify the learning rate based on the number of sentences in each bag as follows:  i =    , if | s i | <  1    1 | s i | , if  1  | s i |   2   (  1 | s i | ) 2 , otherwise where  i is the learning rate for i th training entity pair and  1 /  2 are two tunable bag-size thresholds. In Table 3 and Table 4, we see that this strategy significantly improves performance, especially when training on the larger NYTFB -280 K dataset. We also experimented with this method for PCNN+ATT, but found that its performance did not improve. In Section 2, we presented an approach that combines the benefits of PCNN representations and structured learning with latent variables for minimally supervised relation extraction. In this section we present the details of our evaluation methodology and experimental results. Datasets: We evaluate our models on the NYT-Freebase dataset (Riedel et al., 2010) which was created by aligning relational facts from Freebase with the New York Times corpus, and has been used in a broad range of prior work on minimally supervised relation extraction. Several versions of this dataset have been used in prior work; to facilitate the reproduction of prior results, we experiment with two versions of the dataset used by Riedel et. al. (2010) (henceforth NYTFB -68 K ) and Lin et. al. (2016) (NYTFB -280 K ). Statistics of these datasets are presented in Table 8. A more detailed discussion about the differences between Dataset NYTFB -68 KNYTFB -280 K (Riedel et. al. 2010) (Lin et. al. 2016) Entity pairs 67,946 280,275 Sentences 120,290 523,312 Table 1: Number of entity pairs and sentences in the training portion of Riedel's HELDOUT dataset (NYTFB -68 K ) and Lin's dataset (NYTFB -280 K ). datasets used in prior work is also presented in Appendix B. Hyperparameters: Following Lin et. al. (2016), we utilize word embeddings pre-trained on the NYT corpus using the word2vec tool, other parameters are initialized using the method described by Glorot and Bengio (2010). The Hoffmann et. al. sentential evaluation dataset is split into a development and test set and grid search on the development set was used to determine optimal values for the learning rate  among { 0 . 001 , 0 . 01 } , KB disagreement penalty scalar  among { 100 , 200 ,    , 2000 } and  1 /  2 bag size threshold for the adaptive learning rate among { 10 , 15 ,    , 40 } . Other hyperparameters with fixed values are presented in Table 2. Neural Baselines: To demonstrate the effectiveness of the our approach, we compare against col-less universal schema (Verga et al., 2016) in addition to the PCNN+ATT model of Lin et. al. (2016). After training the Lin et. al. model to predict observed facts in the KB, we use its attention layer to make mention-level predictions as follows: p ( r j | x i ) = exp ( r j  x i ) (cid:80) n r k =1 exp ( r k  x i ) Where r j indicates the vector representation of the j th relation. Structured Baselines: In addition to initializing convolutional filters used in the  PCNN (  ) factors randomly and performing structured learning of representations as in Equation 4, we also experimented with variants of MultiR and DNMAR, which are based on the structured perceptron (Collins, 2002), using fixed sentence representations: both traditional sparse feature representations, in addition to pre-trained continuous representations generated using our best-performing reimplementation of PCNN+ATT. For the structured perceptron baselines, we also experimented with variants based on MIRA (Crammer and Singer, 2003), which we found to provide consistent improvements. More details are provided in Appendix A. 3.1 Sentential Evaluation In this work, we are primarily interested in mention-level relation extraction. For our first set of experiments (Tables 3 and 4), we use the manually annotated dataset created by (Hoffmann et al., 2011). Note that sentences in the Hoffman et. al. dataset were selected from the output of systems used in their evaluation, so it is possible there are high confidence predictions made by our systems that are not present. Therefore, we further validate our findings, by performing a manual inspection of the highest confidence predictions in Table 5. NYTFB -68 K Results: As illustrated in Table 3, simply applying structured models (MultiR and DNMAR) with pre-trained sentence representations performs competitively. MIRA provides consistent improvements for both sparse and dense representations. PCNN+ATT outperforms most latent-variable models on the sentential evaluation, we found this result to be surprising as the model was designed for extracting proposition-level facts. Col-less universal schema does not perform very well in this evaluation; this is likely due to the fact that it was developed for the KBP slot filling evaluation (Ji et al., 2010), and only uses the part of a sentence between two entities as an input representation, which can remove important context. Our proposed model, which jointly learns sentence representations using a structured latent-variable model that allows for the possiblity of missing data, achieves the best overall performance; its improvements over all baselines were found to be statistically significant according to a paired bootstrap test (Efron and Tibshirani, 1994; Berg-Kirkpatrick et al., 2012). 2 NYTFB -280 K Results: When training on the larger dataset provided by Lin et. al. (2016), linguistic features are not available, so only neural representations are included in our evaluation. As illustrated in Table 4, PCNNNMAR also achieves the best performance when training on the larger 2 p-value is less than 0.05. The AUC of most models decreases on the Hoffmann et. al. sentential dataset when training on NYTFB -280 K . This is not surprising, because the Hoffmann et. al. dataset is built by sampling sentences from positive predictions of models trained on NYTFB 68 K ; changing the training data causes a difference in the ranking of high-confidence predictions for each model, leading to the observed decline in performance against the Hoffmann et. al. dataset. To further validate our findings, we also manually inspect the models' top predictions as described below. Manual Evaluation: Because the Hoffmann et. al. sentential dataset does not contain the highest confidence predictions, we also manually inspected each model's top 500 predictions for the most frequent 4 relations, and report precision @ N to further validate our results. As shown in Table 5, for NYTFB -68 K , PCNN+ATT performs comparably on /location/contains 3 and /person/company , whereas our model has a considerable advantage on the other two relations. For NYTFB -280 K , our model performs consistently better on all four relations compared with PCNN+ATT. When training on the larger NYTFB 280 K dataset, we observe trend of increasing mention-level P@N for PCNNNMAR , however the performance of PCNN+ATT appears to decrease. We investigate this phenomenon further below. Performance at Extracting New Facts: To explain PCNN+ATT's drop in mention-level perfor-3 /location/contains is the most frequent relation in the Hoffmann et. mance after training on the larger NYTFB -280 K dataset, our hypothesis is that the larger KB-supervised dataset not only contains more true positive training examples but also more false negative examples. This biases models toward predicting facts about popular entities, which are likely to exist in Freebase. To provide evidence in support of this hypothesis, we divide the manually annotated dataset from Hoffmann et. al. into two categories: mentions of facts found in Freebase, and those that are not; this distribution is presented in the Table 6. In Table 7, we present a breakdown of model performance on these two subsets. For PCNN+ATT, although the AUC of in-Freebase mentions on the test set increases after training on the larger NYTFB -280 K , its Out-Of-Freebase AUC on both dev and test sets drops significantly, which clearly illustrates the problem of increasing false negatives during training. In contrast, our model, which explicitly allows for the possibility of missing data in the KB during learning, has relatively stable performance on the two types of mentions, as the amount of weakly-supervised training data is increased. In Section 3.1, we evaluated the results of minimally supervised approaches to relation extraction by comparing extracted mentions against human judgments. An alternative approach, which has been used in prior work, is to evaluate a model's performance by comparing predictions against held out facts from a KB. Taken in isolation, this approach to evaluation can be misleading, because it penalizes models that extract many Model Name DEV TEST Fixed Sentence Representations MultiR continuous 72.4 66.7 MultiR continuous MIRA 74.6 73.4 DNMAR continuous 73.1 68.0 DNMAR continuous MIRA 75.6 68.7 Jointly Learned Representations PCNNNMAR 78.1 75.4 PCNNNMAR (bag size adaptive learning rate) 82.9 83.1 Baselines col-less universal schema (Verga et al., 2016) 60.3 57.5 PCNN+ATT (Lin et al. (2016) code) 67.9 72.1 PCNN+ATT (our reimplementation with parameter tuning) 78.2 74.8 Table 4: AUC of sentential evaluation precision / recall curves for all models trained on NYTFB -280 K . new facts that do not already appear in the knowledge base. This is undesirable, because the whole point of an information extraction system is to ex-Model Dataset InFB OutFB DEV PCNN+ATT NYTFB -68 K 78.2 89.6 NYTFB -280 K 77.1 77.0 Change -1.1 -12.6 PCNNNMAR (AdapLR) NYTFB -68 K 81.3 90.4 NYTFB -280 K 77.7 90.6 Change -3.6 +0.2 TEST PCNN+ATT NYTFB -68 K 78.7 75.9 NYTFB -280 K 81.9 56.8 Change +3.2 -19.1 PCNNNMAR (AdapLR) NYTFB -68 K 85.9 85.4 NYTFB -280 K 83.1 81.5 Change -2.8 -3.9 Table 7: Top: Comparison of AUCs of In-Freebase and Out-Of-Freebase mentions on sentential DEV set for PCNN+ATT and PCNNNMAR (AdapLR) with two datasets. tract new facts that are not already contained in a KB. Furthermore, sentential extraction has the benefit of providing clear provenance for extracted facts, which is crucial in many applications. Having mentioned these limitations of the held-out evaluation metrics, however, we now present results using this approach to facilitate comparison to prior work. Figure 2 presents precision-recall curves against held out facts from Freebase comparing PCNNN-MAR to several baselines and Figure 3 presents results on the larger NYTFB -280 K dataset. All models perform better according to the held out evaluation metric when training on the larger dataset, which is consistent with our hypothesis, presented at the end of Section 3.1. Our structured model with learned representations, PCNNNMAR (AdapLR), has lower precision when recall is high. This also fits with our hypothesis, as systems that explicitly model missing data will extract many correct facts that do not appear in the KB, resulting in an under-estimate of precision according to this metric. Knowledge Base Population: There is a long line of prior work on learning to extract relational information from text using minimal supervision. Early work on semantic bootstrapping (Hearst, 1992; Brin, 1998; Agichtein and Gravano, 2000; Carlson et al., 2010; Gupta and Manning, 2014; Qu et al., 2018), applied an iterative procedure to extract lexical patterns and relation instances. These systems tend to suffer from the problem of semantic drift, which motivated work on distant supervision (Craven et al., 1999; Snyder and Barzilay, 2007; Wu and Weld, 2007; Mintz et al., 2009), that explicitly minimizes standard loss functions, against observed facts in a knowledge base. The TAC KBP Knowledge Base Population task was a prominent shared evaluation of relation extraction systems (Ji et al., 2010; Surdeanu, 2013; Surdeanu et al., 2010, 2012). Recent work has explored a variety of new neural network architectures for relation extraction (Wang et al., 2016; Zhang et al., 2017; Yu et al., 2015), experimenting with alternative sentence representations in our framework is an interesting direction for future work. Recent work has also shown improved performance by incorporating supervised training data on the sentence level (Angeli et al., 2014; Beltagy et al., 2018), in contrast our approach does not make use of any sentence-level labels during learning and therefore relies on less human supervision. Finally, prior work has explored a variety of methods to address the issue of noise introduced during distant supervision (Wu et al., 2017; Yaghoobzadeh et al., 2017; Qin et al., 2018). Another line of work has explored open-domain and unsupervised methods for IE (Yao et al., 2011; Ritter et al., 2012; Stanovsky et al., 2015; Huang et al., 2016; Weber et al., 2017). Universal schemas (Riedel et al., 2013) combine aspects of minimally supervised and unsupervised approaches to knowledge-base completion by applying matrix factorization techniques to multi-relational data (Nickel et al., 2011; Bordes et al., 2013; Chang et al., 2014). Rows of the matrix typically model pairs of entities, and columns represent relations or syntactic patterns (i.e., syntactic dependency paths observed between the entities). Representations: Prior work has investigated the combination of structured learning with learned representations for a number of NLP tasks, including parsing (Weiss et al., 2015; Durrett and Klein, 2015; Andor et al., 2016), named entity recognition (Cherry and Guo, 2015; Ma and Hovy, 2016; Lample et al., 2016) and stance detection (Li et al., 2018). We are not aware of any previous work that has explored this direction on the task of minimally supervised relation extraction; we believe structured learning is particularly crucial when learning from minimal supervision to help address the issues of missing data and overlapping relations. 3065 5 Conclusions In this paper we presented a hybrid approach to minimally supervised relation extraction that combines the benefits of structured learning and learned representations. In International Conference on Machine Learning , pages 983992. Iz Beltagy, Kyle Lo, and Waleed Ammar. 2018. Improving distant supervision with maxpooled attention and sentence-level supervision. arXiv preprint arXiv:1810.12956 . Sergey Brin. 1998. Extracting patterns and relations from the world wide web. In International Workshop on The World Wide Web and Databases , pages 172183. Springer. Mark Craven, Johan Kumlien, et al. 1999. Constructing biological knowledge bases by extracting information from text sources. In ISMB , volume 1999, pages 7786. Extensive experiments show that by performing inference during the learning procedure to address the issue of noise in distant supervision, our proposed model achieves state-of-the-art performance on minimally supervised mention-level relation extraction. Funding was provided by the National Science Foundation under Grant No. IIS-1464128, the Defense Advanced Research Projects Agency (DARPA) via the U.S. Army Research Office (ARO) and under Contract Number W911NF-17-C-0095 and the Office of the Director of National Intelligence (ODNI) and Intelligence Advanced Research Projects Activity (IARPA) via the Air Force Research Laboratory (AFRL) contract number FA8750-16-C0114, in addition to an Amazon Research Award and an NVIDIA GPU grant. The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation here on.
method method result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain method objective abstain method other abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method other other other abstain abstain abstain other abstain abstain abstain other abstain method abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain other abstain abstain result abstain abstain abstain result other other other other abstain abstain other other other other other abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other