Open Information Extraction (OpenIE) is the task of extracting (subject, predicate, object) triples from natural language sentences. Current OpenIE systems extract all triple slots independently. In contrast, we explore the hypothesis that it may be beneficial to extract triple slots iteratively: first extract easy slots, followed by the difficult ones by conditioning on the easy slots, and therefore achieve a better overall extraction. Based on this hypothesis, we propose a neural OpenIE system, MILIE, that operates in an iterative fashion. Due to the iterative nature, the system is also modularit is possible to seamlessly integrate rule based extraction systems with a neural end-to-end system, thereby allowing rule based systems to supply extraction slots which MILIE can leverage for extracting the remaining slots. We confirm our hypothesis empirically: MILIE outperforms SOTA systems on multiple languages ranging from Chinese to Arabic. Additionally, we are the first to provide an OpenIE test dataset for Arabic and Galician. Open Information Extraction (OpenIE) aims to extract structured facts in the form of (subject, relation, object) -triples from natural language sentences (Etzioni et al., 2008). For example, given a sentence, "Barrack Obama became the US President in the year 2008" , an OpenIE system is expected to extract the following triples: ( Barrack Obama ; became ; US President ) and ( Barrack Obama ; became US President in ; 2008 ). We refer to subject, predicate and the object of the triple as slots of a triple. OpenIE extractions are schema-free, human understandable intermediate representations of facts in source texts (Mausam, 2016). They are useful in a variety of information extraction end tasks such as summarization (Xu and Lap-ata, 2021), question answering (Khot et al., 2017; Yan et al., 2018) and automated schema extraction (Nimishakavi et al., 2016). The various slots of a triple are dependent on each other and hence an error in one slot renders the entire extraction unusable. We hypothesize that triple extraction errors largely stem from the difficulty of extracting certain slots of a triple and said difficulty may depend on the sentence construction and the language. For example, "Barrack Obama became the US President in the year 2008" contains two triples (Barrack Obama; became; US President) and (Barrack Obama; became US President in; 2008) . Extracting the predicate, "became US President in" , for the second triple is tricky, because the object of the first triple (US President) overlaps with the predicate of the second triple. But if the extraction system was provided with the object, (2008) , and then asked to extract a triple conditioned on this object, the predicate extraction would be easier. This is precisely the hypothesis we wish to investigate is it easier to extract certain slots of a triple, say subjects, compared to other slots, such as objects, and is it possible to improve performance by leveraging specific slot extraction orders? Given the hypothesis, we propose MILIE, a M odular & I terative multi L ingual open Information Extraction system, which iteratively extracts the different slots of a triple. The iterative nature allows for (1) studying the effects of a slot extractions on the remaining extractions, (2) extracting easier triple slots followed by harder ones, (3) aggregating different slot extraction orders as a mixture of experts, and (4) integrating slots supplied by an external rule-based system, resulting in a hybrid system. The latter offers a system that combines the best of neural and rule based systems, e.g. by using a rule-based system to extract high precision slots on which the neural system is conditioned. systems. It proves especially useful for zero-shot multilingual extraction, which we evaluated on five different low resource languages. Additionally we show how MILIE can leverage rule-based slot extraction by conditioning on them to predict the remaining parts of the triple. Therefore MILIE is a boon for existing applications wishing to transition from a rule based information extraction system to a neural one, because MILIE would allow using the rule-based system to compensate for the lack of exhaustive training data. Finally, we perform linguistic analyses that uncovers useful insights on how different languages either make it easy or difficult for OpenIE systems to extract individual elements of the triple. Our contributions are summarized as follows: 1. We propose MILIE, a multilingual OpenIE system that iteratively extracts the different slots of a triple. 2. We carry out extensive experiments on a variety of languages (English, Chinese, German, Arabic, Galician, Spanish and Portuguese) and demonstrate that MILIE outperforms re-cent SOTA systems by a wide margin, especially on languages other than English. 3. We perform an extensive analysis based on ablation studies and uncover interesting insights about the nature of OpenIE task in different languages. The backbone of our system is the iterative procedure (Section 2.1), which allows us to investigate our hypothesis. The iterative procedure allows us to extract triple slots in various pathway orders, which results in a series of possible aggregation schemes (Section 2.2). To create a strong iterative system, the training paradigm (Section 2.3 needs to consider two aspects: (1) it needs to prepare incomplete triple extractions which represent incomplete triple extractions the system is expected to predict; (2) it creates negative samples that allow for teaching the system when to not continue with an extraction due to a prior error. With the iterative nature we also integrate rule-based systems (Sec-tion 2.4) as well as elegantly handle the specific case of n-ary extractions, where more than 3 slots need to be extracted (Section 2.5). On top of this block, we add a total of four neural networks blocks in parallel, which we refer to as heads and which are each in charge of extracting a particular triple slot. Concretely, we have the heads f s , f o , f p , f a , which are in charge of predicting s ubject, o bject, p redicate and a rgument, respectively. The argument head is an extra feature, which is needed for n-ary extractions that occur in some datasets, where in addition to the triple there might be an argument that modifies the triple, e.g., a temporal phrase. Given an input sequence of words of length N , S = w 1 ,    , w N , the task for each extraction head is framed as a BIO tagging problem. For this, each output head outputs a label l i for token w i , where l i  { B, I, O } , i = 1    N ( see Figure 1 for the architecture). The output heads use the final transformer hidden state and predict labels denoted by L s , L o , L p , L a where L (  ) = l 1 , l 2 ,    l N . By having different extraction heads, we identify extraction slots iteratively. time, along with the input sentence, the model also expects extractions predicted by the previous iterations. To provide this information we add special symbols to the sentence that explicitly mark the previous extractions in the sentence. For example, we surround the predicate with the symbol <P> , subject with <S> and object with <O> . For example, for predicting the object given the predicate extracted from previous iteration, the extracted predicate is marked in the sentence using the <P> symbol and the sentence is consequently passed through the transformer for predicting the object using the object head. We always extract the arguments at the last iteration, therefore we do not mark the arguments in the sentence. 1 Finally, we add the option to attach a dependency tag t i to each word w i in the sequence. This additional information may allow the system to more effectively learn how to extract triples. We use a language specific dependency tagger for obtaining the tags. We target languages, which are low resource for OpenIE, but could be high resource for other tasks, such as PoS tagging or dependency parsing. For a graphical overview of the MILIE architecture, see Figure 1. 1 Preliminary experiments suggested that predicted the argument last leads to better overall results. This makes sense intuitively, as the argument can modify the entire triple. The order in which the different triple parts are extracted can be varied. This allows us to investigate the challenge of extracting triple elements in specific order on different languages. Additionally different pathways aid different kinds of extractions and combining them results in a richer set of extractions. Choosing a particular order defines a decoding pathway P uvxy as a sequence of output heads where u, v, x, y  { s, p, o, a } . For example, the decoding pathway P spoa denotes a sequence of output functions ( f s , f p , f o , f a ) . Fixing the n-ary argument extraction in the final iteration we obtain the following six decoding pathwaysP spoa , P sopa , P psoa , P posa , P ospa , P opsa . Let's assume the decoding pathway P psoa : predicates are extracted first, then for each predicate, subjects are extracted, then for each (predicate, subject) pair objects are extracted and finally for every extracted (predicate, subject, object) tuple all the n-ary arguments are extracted. This extraction procedure preserves the relationships between the extracted elements resulting in correctly extracting multiple triples. Figure 2 illustrates this procedure. We hypothesize that some triples are easier to predict if, e.g., the predicate is extracted first while for others subject first would work well. This could differ from triple to triple, but also with different languages. Consequently, some decoding pathways might be more error prone than others. This leads to two questions: (1) Which pathways are best? (2) Can we improve recall by aggregating triples using different decoding pathways? We propose a simple algorithm we term as Water Filling (WF) for aggregating the extractions. This is inspired by the power allocation problem in the communication engineering literature (Ku-mar et al., 2008). Imagine a thirsty person with access to different pots of water with varying levels of purity and with the caveat that the amount of water is inversely proportional to the purity. The natural solution is to first drink the high purity water and move on to the pots in decreasing level of purity until the thirst is quenched. We use the same idea. Treating each decoding pathways as an expert, we assume that the triples extracted by all 6 pathways are more accurate compared to those extracted by only 5 pathways, 4 pathways and so on. This can be thought of as triples obtaining votes from experts. Starting with an empty set, for each sentence we start adding triples to the set in the 6941 order of decreasing number of received votes. The normalized votes a triple receives is used as the confidence value of the triple. Although the procedure is explained in a sequential manner it can be parallelized by running all 6 pathways in parallel. Triple preparation. For effectively extracting different triple slots conditioned on other slots, the model needs to see such combinations during training. However, enumerating all possible combinations exhaustively is prohibitively expensive. We propose a sampling technique that ensures that the model sees varied combinations of different targets and prior extractions. This is done by creating a training set that simulates a prior extraction and forces the model to predict the next extraction. To ensure that the training dataset size does not explode, we randomly sample one pathway order for each training instance. Based on the sampled pathway, we randomly sample at which step in the decoding process we are at and then mark the slots prior to this step in the sentence and use the remaining steps as target labels. We allow for multiple instances of the target labels, however there is only one instance of the marked element. For example, given one subject the target could be multiple predicates. This procedure trains the model to predict an appropriate label conditioned on a variety of previous predictions. At each time step we update the parameters of the currently used head and the underlying model. Given that triples are at different steps in their decoding process, we minimize different log-likelihood functions. We describe the log likelihood functions along with a few example of the training instances in Table 1. We list additional details in Appendix A. Negative Sampling. Iterative prediction is prone to error amplification, i.e. if an error is made during the first iteration then the error propagates and affects subsequent extractions. Anticipating this, we train MILIE to recognize extraction errors made in the previous iteration. We purposely augment the training data with corrupted data points containing incorrectly marked extractions. For each of the incorrect extractions the model is trained to predict a blank extraction, i.e., predicting the outside label for all tokens. We use a similar sampling procedure as described previously. For every training data point from a fixed number of training data points, we create one negative sample using one of the three techniques and then choose k negative samples, where k is a hyperparameter. We corrupt triples using three techniques: (1) corrupting the predicates by replacing them with randomly chosen tokens from the sentence, (2) corrupting the subject and object by exchanging them, and (3) by mismatching the subject object pairs from different triples. We detail the entire procedure in Appendix A. 2.4 Integrating Linguistic Rule based systems Crucially, each output head is conditioned on the input and the output labels extracted by the previous function. This feature allows MILIE to seamlessly integrate rule based systems with neural systems since the conditioning can be also done on extractions obtained from rule based systems. This is advantageous in situations where a linguistic rule based system works well, for say, extracting objects. Then MILIE can complete the missing parts of the triple conditioned on the objects. We treat the output of the rule based system as potential objects paired with subjects and extract the predicate connecting them. If the rule based extraction is incorrect, then MILIE can detect the error and extract nothing. This results in more accurate extractions compared to simply post-processing the extracted tokens using linguistic rules. We evaluate MILIE on both n-ary as well as binary triple extraction datasets. One simple way to convert the n-ary extractions to binary extraction is to ignore the n-ary arguments. However, this will lead to a decrease in recall because the n-ary arguments may not be part of other extracted triples due to the initial n-ary extraction. Another method is to treat the extracted n-ary arguments as objects to the same subject, predicate pair. This would ensure that the extracted arguments are not dropped, however this may result in drop of precision since the n-ary argument may not attach to the same predicate. For example, consider the extraction ( Barrack Obama; became; US President; in the year 2008 ). Treating n-ary arguments as objects results in ( Barrack Obama; became; US President ) and ( Barrack Obama; became; in the year 2008 ) resulting in an incorrect extraction. In contrast to the above subpar solutions, the iterative nature of MILIE allows us to elegantly address the problem of converting n-ary extractions into a 6942 Likelihood function Input Sentence Head Target L p =  (cid:80) Ni =1 log p ( l pi | f p (  ); S ) The Taj Mahal was built by Shah Jahan in 1643 Predicate built by L s =  (cid:80) Ni =1 log p ( l si | f s (  ); S ; L p ) The Taj Mahal was <P>built by<P> Shah Jahan in 1643 Subject Taj Mahal L o =  (cid:80) Ni =1 log p ( l oi | f o (  ); S ; L p ; L s ) The <S>Taj Mahal<S> was <P>built by<P> Shah Jahan in 1643 Object Shah Jahan L a =  (cid:80) Ni =1 log p ( l ai | f a (  ); S ; L p ; L s ; L o ) The <S>Taj Mahal<S> was <P>built by<P> <O>Shah Jahan<O> in 1643 Argument in 1643 L p =  (cid:80) Ni =1 log p ( l pi | f p (  ); S ; L s ; L o ) The <S>Taj Mahal<S> was built by <O>Shah Jahan<O> in 1643. binary format: we treat the extracted n-ary arguments as hypothesized objects. We then provide the extracted subject, hypothesized object pair to the model, which then extracts a new predicate conditioned on the previously extracted subject and the hypothesized object, i.e., p ( L p | f p (  ); S ; L s = "Barrack Obama" ; L o = "year 2008" ) . This creates a possibility of extracting the correct predicate, something that is not possible with existing n-ary OpenIE systems. Baselines & Training. We compare MILIE with both unsupervised and supervised baselines. Specifically we compare MILIE with ClausIE, MinIE, Stanford-OIE, RNN-OIE, OIE6 (Del Corro and Gemulla, 2013; Gashteovski et al., 2017; Stanovsky et al., 2018; Angeli et al., 2015; Kolluru et al., 2020a) and Multi2OIE (Ro et al., 2020) on English. Multi2OIE is the only neural system capable of extracting triples from multiple languages and therefore it is the only available baseline for the non-English evaluations. We use the English RE-OIE2016 (Zhan and Zhao, 2020) training dataset used in (Ro et al., 2020). This training dataset contains n-ary extractions allowing MILIE to be evaluated on both n-ary as well as binary extraction benchmarks. Evaluation on languages other than English is always zero-shot, i.e., the model is trained using only the English Re-OIE2016 dataset and tested on test set of the other languages. CaRB benchmark. We use the CARB benchmark introduced in (Bhardwaj et al., 2019) for evaluating English OpenIE n-ary extraction. However, the CARB benchmark also suffers from serious shortcomings due to its evaluation method based on token overlaps. For example, (Gashteovski et al., 2021) discovered that a simple OpenIE system that breaks the sentence into a triple at the verb boundary achieves 0 . 70 recall and 0 . 19 precision. This is problematic since it indicates that simply adding extraneous words to the extraction results in improved recall. BenchIE benchmark. Due to the issues identi-fied for CaRB, we also evaluate using BenchIE, which is an exhaustive fact based multilingual OpenIE benchmark proposed by (Gashteovski et al., 2021). BenchIE evaluates explicit binary extractions in English, Chinese, and German. BenchIE is accompanied by an annotation tool, AnnIE (Friedrich et al., 2021), for extending the benchmark to additional languages. For Arabic, we translated 100 sentences from BenchIE-English to Arabic with the help of a native Arabic speaker and then extracted triples using AnnIE. Similarly for Galician we translated all 300 sentences to Galician 6943 Chinese German Arabic Galician F1 P R F1 P R F1 P R F1 P R M2OIE 17.1 25.7 12.8 4.0 8.9 2.6 4.9 16.3 2.9 8.7 14.7 6.2 milIE 20.5 25.2 17.3 8.5 13.4 6.3    18.3 23.7 14.8 DEP 19.2 19.8 18.7 8.4 11.3 6.7 7.3 14.2 4.9 13.9 16.6 11.9 NS 17.3 19.6 15.5 10.3 14.3 8.0 4.0 10.8 2.5 13.7 18.5 10.9 Bin 20.0 22.0 18.4 9.0 13.5 6.7 7.5 13.8 5.1 17.3 21.7 14.4 Table 3: MILLIE performance comparison on multilingual BenchIE. Multilingual CaRB. Additionally we also evaluate MILIE on the Spanish and Portuguese multilingual CaRB datasets introduced in Ro et al. (2020). The lexical match evaluation used in this dataset has numerous shortcomings (Bhard-waj et al., 2019), however we include it for a fair comparison to Ro et al. (2020)'s Multi2OIE system. The CARB test set was translated to Spanish and Portuguese using the Google Translate API. To investigate the quality of these automatic translations, we randomly sampled 100 sentences from the test sets and had them evaluated by native Spanish and Portuguese speakers. To our surprise we discovered that around 70 percent of the sentence or extraction translations were inaccurate. Table 2 shows a few examples of the incorrect translations. For an accurate and clean comparison with Multi2OIE we also cleaned up part of the Spanish test set by re-translating 149 sentences and their extractions in Spanish. These translations were done by native Spanish speakers. On the CARB English benchmark we use results for baselines reported in (Ro et al., 2020) and (Kolluru et al., 2020a). For evaluating on BenchIE, we run all the baselines on the BenchIE English evaluation benchmark. For multilingual BenchIE we train Multi2OIE using the code and hyperparameters supplied in the paper. For hyperparameter tuning we use the CARB English validation set and use the F1 scores obtained using the CARB evaluation procedure for comparing models with different hyperparameters. The MILIE model is trained using negative sampling and includes the dependency tag information and binarization. We use the spaCy dependency parser for obtaining dependency tags. We were unable to find a dependency parsing tool with universal dependencies for Arabic and therefore we did not use dependency tags for Arabic. For BenchIE, MILIE uses the binarization function described in Section 2.5, but not for CARB and lexical match because they evaluate n-ary extractions. In Table 5, we compare MILIE with several unsupervised and supervised baselines in English on CARB and BenchIE. MILIE performs much better compared to other neural baselines on BenchIE. This is not the case for the CARB dataset since CARB penalizes compact extractions and rewards longer extractions (Gashteovski et al., 2021). Although rule based systems like ClausIE and MinIE outperform neural systems, they cannot be used for languages other than English. In Table 3, we compare MILIE with Multi2OIE (M2OIE) on the multilingual BenchIE benchmark. MILIE performs significantly better compared to Multi2OIE for all the languages. For German and Arabic both Multi2OIE and MILIE perform significantly worse compared to the other languages. The presence of separable prefixes in German verbs which cannot be extracted using BIO tags results in low performance. The BIO tagging scheme assumes continuity of phrases which is absent for most German verbs present in predicates, resulting in extremely low recall. For Arabic, the low scores are due to the Verb-Subject-Object nature of the Arabic language along with the fact that subjects or objects can be expressed as part of the verb. This calls for additional research on framing OpenIE tasks for languages such as German and Arabic. MILIE significantly outperforms Multi2OIE for Galician language which is closely related to Portuguese. Ablation results in Table 3 also indicate the usefulness of adding the dependency tags, negative sampling, and the binarization mechanism. In Table 4, we compare MILIE with Multi2OIE on the CARB lexical match benchmark. MILIE, without negative sampling works best for Spanish clean data. This is not due to the language, but due to the lexical match evaluation which rewards overly long extractions even if incorrect. Not using negative sampling sometimes improves recall which may improve F1 score. This is observed for the German benchmark. MILIE can easily integrate any rule based system that extracts even a part of the triple. To evaluate this, we first simulate a system that only extracts the object and use MILIE to extract other parts of the triple. We do this by employing ClausIE for extracting triples for the BenchIE English data and only use the object, discarding the rest of the triple. The reason behind the choice of selecting object extraction from ClausIE is the fact that neural systems are not good at extracting objects (Kolluru et al., 2020a). This is also seen from additional experiments detailed in Section 4. Table 6 indeed confirms that combining rule based object extraction with MILIE improves performance by over 6% in F1 score. This showcases that MIL IE's ability to integrate other systems can be a great advantage. We would like to analyze that the ability of MILIE to extract triples using different extraction patterns results in improved performance on multilingual data. For this, we compare MILIE with the water filling aggregation against MILIE with different extraction pathways.We also compared with a dynamic decoding scheme where MILIE chooses a decoding pathways based on the sentence. To do this we split a part of the English training set and for each sentence in the split we record the extraction pathway that provides the best F1 score MILIE 6945 Figure 3: Percentage error contribution due to incorrect subject, predicate or object for EN, DE, ZH and AR. as per CARB evaluation. We then use this as training data for training another mBERT model which classifies each sentence in one of the six classes where each class represents an extraction pathway. Table 7 details the performance for different extraction schemes. All the extraction schemes except WF, use only one pathway. DYN provides mixed results across the different languages for German it is the best approach, whereas for Arabic it is the worst. In contrast, the combination of multiple pathways allows to performing much better than the other approaches on all languages, except German. This demonstrates that combining triple extraction from multiple pathways is better than any single pathway, which in turn confirms that extracting triples repeatedly from the same sentence using multiple extraction pathways is more profitable than using a single extraction pathway. Additionally, Table 7 provides an interesting insight: predicate first seems to be the best, followed by subject first and then object first for languages other than Arabic. This also shows how the difficulty of extracting triple slots using transfer learning from English varies with the target language. Table 7 suggests that predicates are easier to extract leading to lesser number of errors propagated in the prediction chain. We suspect that this could result from differences in linguistic variability. To test our hypothesis we measured the entropy of the distribution of dependency and part-of-speech tags in the predicate, subject and object slots in the BenchIE English and the multilingual test sets. Results shown in Table 8 suggest that linguistic complexity of objects is higher than those of predicates and subjects. This is also confirmed in Figure 3, where we plot the extraction errors in either subject, predicate or objects among incorrectly extracted triples. Most errors result from extracting incorrect objects compared to predicates and subjects. The percentage sum does not add to hundred because an incorrect triple can contain errors in more than one slot. OpenIE systems largely come in two flavors, (1) unsupervised OpenIE systems that use fine grained rules based on dependency parse trees (Del Corro and Gemulla, 2013; Gashteovski et al., 2017; Lauscher et al., 2019), and (2) supervised neural OpenIE systems, trained end-to-end with large training datasets (Stanovsky et al., 2018; Ro et al., 2020; Kolluru et al., 2020a). Neural OpenIE systems characterize OpenIE as either a sequence tagging task (Stanovsky and Dagan, 2016; Ro et al., 2020), span prediction task or a sequence generation task (Kolluru et al., 2020b). However all these prior approaches extract a triple in a single step, which does not allow us to study the effect of extracting a specific slot and its effect on extracting the rest of the triple. Neural generative approaches to OpenIE use sequence-to-sequence models with a copy mechanism for generating triples (Sun et al., 2018; Kolluru et al., 2020b). The copy mechanism needs to be learned and is often a source of errors. A series of alternative approaches cast OpenIE as a sequence tagging task where each token is tagged as subject, predicate or object using a BIO like tagging scheme (Stanovsky et al., 2018; Ro et al., 2020; Kolluru et al., 2020a). In these systems, all triple slots are extracted simultaneously and it is therefore not possible to condition on easier slots. More closely related to our work is SpanOIE (Zhan and Zhao, 2020) and Multi2OIE (Ro et al., 2020), which first extracts the predicate and then 6946 all additional arguments. Like us, Multi2OIE (Ro et al., 2020) addresses multilinguality by leveraging a pretrained BERT model (Devlin et al., 2019) for transfer learning. In contrast, through our iterative nature, it is possible to enrich the extractions in other languages if rule based models or other models (e.g. NER recognizers) exist to provide input for a triple slot. IMOJIE (Kolluru et al., 2020b) iteratively extracts entire triples from a sentence: first a triple is extracted, which is added to the input to extract the next triple. In contrast, our work iteratively extracts the slots of a single triple, which allows us to condition on the easier slots and therefore obtain higher quality triples. (Kolluru et al., 2020a) propose OpenIE6, a BERT based system, with iterative grid labelling and linguistic constraint based training. Such lingusitic constraints with soft penalties cannot be readily ported to other languages since such constraints use head verb based heuristics. Consequently OIE 6 is evaluated only on English. We introduced MILIE, a modular & iterative multilingual OpenIE system. We confirmed our hypothesis that it is beneficial to extract triple slots iteratively which allows us to extract easier slots first. Our experiments on English as well as five low resource languages uncovered that, with the exception of Arabic, triples are easier to extract if the predicate is extracted first followed by the subject and object. More importantly we discovered that extracting triples using multiple extraction pathways is superior than the standard single extractions especially in the multilingual setting. We also demonstrated how MILIE can be combined seamlessly with rule based systems for improving performance. Although our experiments were focused on the OpenIE task, we believe that the insights gained can be translated to other information extraction tasks with coupled extractions. We plan to explore such connections in the future.
abstain abstain objective objective abstain result objective abstain method abstain abstain abstain abstain objective abstain objective abstain objective objective abstain abstain abstain method result abstain objective objective objective objective objective abstain abstain abstain method method method abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method method method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain other method abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain method method abstain method method abstain method abstain method method method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain other other method other other other other abstain abstain objective other objective other other other abstain objective objective method objective method objective