Progress with supervised Open Information Extraction (OpenIE) has been primarily limited to English due to the scarcity of training data in other languages. In this paper, we explore techniques to automatically convert English text for training OpenIE systems in other languages. We introduce the Alignment-Augmented Consistent Translation (AACTRANS) model to translate English sentences and their corresponding extractions consistently with each other  with no changes to vocabulary or semantic meaning which may result from independent translations. Using the data generated with AACTRANS, we train a novel two-stage generative OpenIE model, which we call GEN2OIE, that outputs for each sentence: 1) relations in the first stage and 2) all extractions containing the relation in the second stage. GEN2OIE increases relation coverage using a training data transformation technique that is generalizable to multiple languages, in contrast to existing models that use an English-specific training loss. Evaluations on 5 languages  Spanish, Portuguese, Chinese, Hindi and Telugu  show that the GEN2OIE with AACTRANS data outperforms prior systems by a margin of 6-25% F1. 1 1 Introduction Open Information Extraction (OpenIE) is the task of converting unstructured text to semi-structured tuples of the format < subject; relation; object >, where these three components are textual phrases, broadly extracted from the original text (Etzioni et al., 2011). OpenIE tuples have shown utility in various downstream tasks (Mausam, 2016) like Question Answering (Fader et al., 2013; Khot et al., 2017), Machine Reading (Poon et al., * denotes equal contribution 1 Code and models released at github.com:dair-iitd/moie 2010), Multi-Document Summarization (Chris-tensen et al., 2014; Fan et al., 2019), Schema Induction (Balasubramanian et al., 2013), and Knowledge Base Construction (Gupta et al., 2019; Chandrahas and Talukdar, 2021). With widespread adoption of Deep Learning in NLP, Open Information Extraction (OpenIE) systems have gone through a paradigm shift from using rule-based, statistical systems to supervised neural models. However, both types of systems have been limited to only a few languages  earlier systems required language-specific OpenIE insights, and current systems require annotated training corpus that pose a barrier, particularly for low-resource languages. Related tasks such as Semantic Role Labeling face similar challenges in extending to multiple languages. X-SRL (Daza and Frank, 2020) addresses this by automatic translation of English sentences to the target language followed by label projection to infer the semantic role labels in the translated sentence. However, translating the sentence alone may be insufficient for OpenIE because the generated tuples (also referred to as extractions) can include additional words absent in the sentence or require some changes to the word morphology used in the sentence. Although less prevalent in English, these characteristics need to be addressed in other languages. X-SRL approach may be extended such that each extraction can also be automatically translated and subject, relation, object labels projected from English extractions. However, independent translation of sentence and extraction may introduce unwanted lexical (e.g. synonyms) or semantic (e.g., change in gender) variations between the translations, as shown in Table 1. Such translation inconsistencies in the training data lead to invalid OpenIE examples. lations must use same words or their morphological variants as much as possible. Hence, we propose Alignment-Augmented Consistent Translation (AACTRANS), a seq2seq model that translates the given input text in a way that is consistent with a reference translation by biasing the translation to use words similar to the reference. To ensure that translations of sentence and extractions are consistent with each other, we use AACTRANS model to translate each of them with the same reference. In Section 4.1, we describe the reference used in training and inference. Both generation based (Kolluru et al., 2020b) and labeling based (Ro et al., 2020) architectures have shown competitive performance on English OpenIE. However, labeling based models cannot naturally introduce new words or change morphology of sentence words required in some languages. Therefore, we use a new generative model, GEN2OIE, that contains two stages: the first stage produces all the relations in the sentence and the second stage generates the extractions containing the given relation. We also use a training heuristic specific to two stage models that increases relation coverage across multiple languages. Our major contributions are that we: 1. introduce a novel technique for transferring data from English to other languages using the AACTRANS model and label projection, 2. propose two-stage generative model, GEN2OIE, for training OpenIE system in multiple languages, 3. release OpenIE evaluation datasets for two Indian languages, Hindi and Telugu, and Our work is in line with the recent trend of extending IE and knowledge-based NLP systems to multiple languages. Recent works have explored distantly supervised relation extraction (Rathore et al., 2022; Bhartiya et al., 2022), knowledge-base completion (Singh et al., 2021), and fact linking (Kol-luru et al., 2021). Our focus is OpenIE. Many of the prior OpenIE systems, both nonneural (OpenIE-4 (Pal and Mausam, 2016; Christensen et al., 2011), OpenIE-5 (Saha et al., 2017; Saha and Mausam, 2018), ClausIE (Del Corro and Gemulla, 2013)) and neural (RnnOIE (Stanovsky et al., 2018), OpenIE-6 (Kolluru et al., 2020a)) have been deployed for English. Moreover, OpenIE systems built for other languages often work only for a single language due to their reliance on language-specific resources. For example, Bassa et al. (2018); Rahat and Talebpour (2018); Ro-madhony et al. (2018); Guarasci et al. (2020); Papadopoulos et al. (2021) focus on German, Persian, Indonesian, Italian, and Greek, respectively. Claro et al. (2019) present the importance of and various challenges involved with building multilingual OpenIE systems. Neural models like Logician (Sun et al., 2018) and CrossOIE (Cabral et al., 2020) use language-specific training data. Reliance on manually-annotated data or language-specific resources makes it infeasible to develop systems for the plurality of languages in the world, due to the cost and effort involved. However, our automated data conversion method can handle even low-resource languages like Telugu. Non-neural systems such as PredPatt (White et al., 2016) and ArgOE (Gamallo and Garcia, 2015) work for multiple languages by using CoNLL-X and Universal Dependency parses respectively, to extract predicate-argument structures. Owing to their pipelined nature, their performance is below that of neural systems like Multi 2 OIE (Ro et al., 2020). Multi 2 OIE is a two-stage labeling model that works for English, Spanish and Portuguese. GEN2OIE extends this 2-stage design to the generative paradigm which allows for better modeling of the OpenIE task. The underlying mBERT encoder in Multi 2 OIE allows for cross-lingual generalization across various languages even after training with only English supervised data. However, dependence on zero-shot generalization limits the performance of the model. Two types of methods have been proposed for constraining the outputs of the machine translation systems: 1) altering the decoding algorithm (Hasler et al., 2018), or 2) modifying the training methodology (Chen et al., 2020; Dinu et al., 2019). We follow the second approach for constraining translations by AACTRANS to be consistent to that of a reference sentence. Unlike prior work which focuses on constraining translations of few words, our task requires constraining the entire translation. We make use of awesome-align (Dou and Neubig, 2021a), an unsupervised word alignment technique (Och and Ney, 2003), that outputs the alignment between words in sentences of two languages. Awesome-align is trained using only parallel set of sentences in the two languages and generates aligned target words for each source word. Transferring linguistic annotations from source to target language has been pioneered by (David et al., 2001) and has been used in context of Semantic Role Labeling (Annesi and Basili, 2010) and PoS-tagging (Zennaki et al., 2019). After consistent translation, we make use of Crosslingual Projection (Faruqui, 2015), to transfer OpenIE tags. For the transfer of OpenIE data from one language to another, we represent the source language 2 as E and the target language as F . Further, we use sent E and ext E to represent a sentence and extraction in the source language and aact sent F and aact ext F to represent the transferred sentence and extraction in the target language. To aid in the translation of extractions, we create a sub-sentence from each extraction by concatenating the phrases in all the fields of the extraction. The order of concatenation is such that the formed sub-sentence is grammatically valid. We refer to this sub-sentence as an ext-sentence and represent it as es L , where the subscript L represents its language. For most English extractions, the ext-sentence corresponds to concatenating the fields in the order of subject, relation and object. However, other languages may follow a different order or allow for multiple orders. We rely on the output of system that translates the English ext-sentence to determine the ext-sentence in other languages. Moreover, each extraction can be seen as a labeling over the words of ext-sentence with either the S ubject, R elation or O bject tags. Tags for each word in the ext-sentence can also be regarded as the extraction. In this section we describe the technique used to convert OpenIE training data from source language E to a target language F . The source sentence, sent E , and all its corresponding ext-sentences, es E , are consistently translated to language F (Section 4.1), and then, for each extraction in language E , ext E , the S , R or O labels are projected to the translated ext-sentence, es F , to form the extraction, ext F , in language F (Section 4.2). Figure 1 describes the pipeline with the help of an example. We introduce a new Seq2Seq-based translation model called Alignment-Augmented Consistent Translation (AACTRANS) to ensure that sentences and ext-sentences are translated consistently from languages E to F . We define two translations as consistent if similar phrases have same grammatical structure, vocabulary and morphology while allowing for minimal changes necessary to ensure fluency. To ensure consistency among translations of multiple pieces of text (both the sentence and respective ext-sentences present in an English OpenIE instance), we make use of a reference text in language F to guide all of their translations. By individually maintaining consistency with the reference, their respective translations end up being consistent to one another as well. To generate a translation f (language F ) of text e (language E ), consistent with a reference r (lan-guage F ), we use the following procedure. Firstly, given e = e 1 e 2 . . . e N and r = r 1 r 2 . . . r M , we find the set of aligned words A e i ={ r j } for each word e i in e , using a word alignment model. Secondly, the aligned text e  is constructed by concatenating each of the words e i in e , with their aligned words A e i , using ## as a separator (shown as <1>, <3>  <4> and <2>, <3>  <5> in Figure 1). If e i is aligned to the words r j , r k ( j < k ), then e  contains e i ## r j r k #. If e i has no aligned words, then e  contains e i #. Thirdly, the AACTRANS model takes e  as input and produces the sequence f as output, which represents a translation of e that is biased to use the aligned reference words (shown as <4>  <7> and <5>  <8> in Figure 1). Next we discuss the training and inference of AACTRANS model. Training : We use parallel sentences of languages E and F that are available in existing translation corpora for training the AACTRANS model. For each parallel sentence pair e and f , we use the sentence f itself as the reference r . Using the alignments between the words of e and f , we form the input e  , as discussed. The AACTRANS Seq2Seq model is trained with e  as input and f as output. Since e  has words from f , the model learns to use them during training. Inference : Here, we consistently translate English sentence sent E and each of its ext-sentences es E . We use an off-the-shelf translation system to translate sent E to language F , represented as t sent F . t sent F is used as the common reference r for constructing aligned sentence al sent EF and aligned ext-sentence al sent EF from sentence sent E and ext-sentence es E , respectively. We then apply the trained AACTRANS model on al sent EF and al sent EF to generate target sentence aact sent F and target ext-sentence aact es F respectively. Each word in the target ext-sentence, aact es F , must be labeled with either the Subject, Relation, or Object tag to form the completed extraction in language F . The tags from the corresponding ext E are projected onto aact es F using the Crosslingual Projection algorithm (Faruqui, 2015) (described in Appendix A), which uses word alignments between es E and aact es F and produces as output, the tags over aact es F , giving extraction aact ext F . The final set of <sentence, extractions> pairs constitute the data for training OpenIE system in language F . Thus the overall flow is: 1) AACTRANS model training is done on parallel corpus, 2) AACTRANS model inference is applied on language E OpenIE examples, 3) CLP projection is used to obtain the labelled extractions, and 4) the generated data is used to train OpenIE system like GEN2OIE, which is discussed next. To train OpenIE systems in multiple languages, we use a novel GEN2OIE model that extends the 2-stage design of Multi 2 OIE (Ro et al., 2020) to a generative paradigm. The first stage generates all possible relations and the second stage generates all extractions that contain a given relation. tion, thus overcoming the limitations of Multi 2 OIE model. Moreover, due to its generative nature, GEN2OIE can add new words or introduce changes in morphology that may be necessary for producing correct extractions, which cannot be achieved by labeling models. Stage-1 Seq2Seq : The input sentence is passed to the encoder and decoder generates a string formed by concatenating the set of relations from all the extractions, separated by an [SEP] token. During training, the target relations are concatenated in the order in which they occur in the sentence. We find that a deterministic order is important for adding stability to the model training. Stage-2 Seq2Seq : To produce extractions corresponding to each relation generated in Stage-1, the relation r is concatenated with the input sentence s and passed to the encoder as  r [SEP] s . The decoder is trained to generate all the extractions containing the relation r . Multiple extractions are separated by an < e > token and each extraction contains delimiters tokens to identify the various parts of the extraction. The surrounding < s >...</ s >, < r >...</ r > and < o >...</ o > tokens are used to identify the subject, relation and object phrases. Labeling models like OpenIE-6 (Kolluru et al., 2020a) have used constrained training to increase the relation coverage. However, the constraints are limited to English and specific to labeling architectures. We introduce a simple parts-of-speech based heuristic during Stage-1 training of GEN2OIE that increases the relation coverage in the generative paradigm while being applicable across languages. Relation Coverage (RC) : We observe that for generating all possible extractions, all the verbs in the sentence must be contained in some relation. However, the extractions of training data may be incomplete and not satisfy this property. Therefore, during the training phase, we modify the input to the Stage-1 model by removing the verbs in the sentence which are not present in relation of any extraction. Thus the model learns that every verb must be included in some relation and applies the same during inference as well. This heuristic does not effect Stage-2 model training. The word log probabilities assigned by the Stage-2 decoder can be summed up to be used as confidence score for the extractions generated by GEN2OIE. We experiment with using a separate model for obtaining the confidence scores. A sequence-labeling model is trained on each lan-guage's extractions with ext-sentence as input and S, R, O labels over the ext-sentence as the output. The log probabilities given by the sequence-labeling model to the labels predicted by the GEN2OIE model are summed up to get the new confidence scores. We train OpenIE systems in 5 languages, Spanish (ES), Portuguese (PT), Chinese (ZH), Hindi (HI) and Telugu (TE), by using the training data transferred from English to the respective language. For training the Seq2Seq models used in the data generation pipeline and the OpenIE systems based on the GEN2OIE architecture, we choose either the mBART (Liu et al., 2020) or mT5 (Xue et al., 2020) model depending on the particular language. Both of them are pre-trained multilingual Seq2Seq models that are trained with a span denoising objective on a large corpus of text containing many languages. mBART is pre-trained on CC25 and mT5 is pre-trained on mC4 corpus which contain text in 25 and 101 languages, respectively. Since mBART does not support Portuguese and Telugu, we use mT5 for these two languages and mBART for the 2506 remaining 3 languages. We use the default hyperparameters recommended for these models and they are reported in Appendix F. Training Datasets : For training the AACTRANS model, we make use of parallel English, language F sentences available in standard translation corpora using the method described in Section 4. For Spanish we use parallel sentences from EuroParl corpus (Koehn et al., 2005), and for Portuguese we use a subset of the ParaCrawl corpus (Ban et al., 2019), as chosen by Lopes et al. (2020). For Hindi we use the IIT-B corpus (Kunchukuttan et al., 2018), and for Telugu we use the Samanantar corpus (Ramesh et al., 2021). For Chinese we use the data released for WMT19 (Barrault et al., 2019). We list the BLEU scores of the various systems in Appendix C. We use the OIE4 training corpus from Kolluru et al. (2020b) and transfer it to other languages for training OpenIE systems. Evaluation Datasets and Metrics : For evaluating translation systems we use the test sets available in the respective corpora and use SacreBLEU (Post, 2018) as the metric. 3 For evaluating different OpenIE systems we use the Optimal F1 and Area Under Curve (AUC) as computed by the CaRB (Bhard-waj et al., 2019) scoring function. For Spanish, Portuguese OpenIE we use test sets provided in Ro et al. (2020). For Chinese OpenIE, we randomly choose 10% of the SAOKE dataset (Sun et al., 2018). In order to evaluate our method on medium and low resource languages, we release new OpenIE test sets in Hindi and Telugu. Human annotators who are fluent in both the language and are knowledgeable about the OpenIE task translated about 300 randomly chosen sentences and their corresponding extractions from CaRB test set. They were paid $2.5 per sentence. Table 2 lists the number of examples in different languages used for training and evaluating translation and OpenIE systems. We perform experiments to answer the questions: 1. How effective is the GEN2OIE model? 2. What is the quality of data generated with the AACTRANS+CLP pipeline, assessed both by 3 BLEU+case.mixed+numrefs.1+smooth.none+tok.intl+version.1.5.1 EN ES PT ZH HI TE TranslationTrain-1.9M 5M 1M 1.6M 4.8M Test 38473 99,087 2001 2507 2390 OpenIETrain91K 91K 91K 91K 91K 91K Test 641 594 594 3833 298 302 Table 2: Data statistics for OpenIE examples and (En-glish, language F ) parallel sentences. the final performance of systems trained using it and with metrics defined for evaluating consistency? 3. What are the roles of different components in the GEN2OIE and AACTRANS+CLP data? To study the baseline monolingual effectiveness of GEN2OIE, we first train and evaluate the system on English data. The results are shown in Table 3. We compare with previously proposed English OpenIE models such as Multi 2 OIE (Ro et al., 2020), OpenIE6 (Kolluru et al., 2020a) and IMoJIE (Kol-luru et al., 2020b). We also consider individual components in OpenIE6, the IGL and Constrained-IGL (CIGL) architectures. CIGL achieves the highest performance among all prior models but uses of English specific constraints in training. We find that GEN2OIE, which uses the proposed language-agnostic relation coverage (RC) outperforms CIGL by 0.4% F1. However, its AUC remains lower. Therefore, we rescore the generated extractions with labeling-based rescoring model (Section 6). This results in a new state of the art for English in F1 and AUC with the labeling-based rescoring resulting in a 2.9% AUC gain over CIGL. To further analyze the effectiveness of our 2-stage architecture, we introduce another model called GENOIE that outputs all extractions for a sentence as a single string, separated by an < e > token. We find that using GENOIE results in (2.3, 2.0)% drop in F1, AUC compared to GEN2OIE which leverages RC. We also report GEN2OIE performance without using RC. In order to test the quality of the OpenIE examples generated using the AACTRANS+CLP pipeline, we train both the GENOIE and GEN2OIE models over the data generated for different languages. In Table 4, we compare it with examples generated from two other methods, SentTrans and SentExtTrans. SentTrans+CLP represents an adaptation of X-SRL (Daza and Frank, 2020) for OpenIE where only the sentence is translated and each extraction, which is expressed as labeling over the words in the sentence, are projected onto the translated sentence using the CLP algorithm described in Section 4.2. The projected extraction is now a labeling over the translated sentence and hence it uses the same morphology as the sentence and cannot add new words. SentExtTrans+CLP uses independent translation of English sentence and ext-sentences followed by CLP algorithm between the English and translated ext-sentences to transfer the labels. Although this allows for adding new words and changing morphology, it can result in a lack of consistency between the translations. We find that both GENOIE and GEN2OIE show consistent gains with AACTRANS+CLP data across various languages, when compared with SentExt-Trans+CLP and SentTrans+CLP data. We further use rescoring models that are trained on the same AACTRANS+CLP data. Labeling-based rescoring achieves significantly higher AUC, with as much as 8.3% gain in Telugu. We experiment with two versions of Multi 2 OIE: 1) trained only on English OpenIE data and applied to other languages in a zero-shot manner and 2) using language-specific training data generated from SentTrans+CLP. We specifically choose Sent-Trans+CLP data as all the extractions can be expressed as labels over the sentence, which is a requirement for training Multi 2 OIE which is itself a labeling model. We find that Multi 2 OIE model trained with SentTrans+CLP data improves over the zero-shot setting in all languages other than Chinese (discussed below). However, it performs significantly worse than GEN2OIE by (5.2, 3.3)% in (F1, AUC) on average, even on training with the same SentTrans+CLP data. This can be attributed to Multi 2 OIE's lack of capability to handle: 1) overlapping relations, 2) multiple extractions per relation, 3) adding auxiliary words or 4) changing inflectional forms, as shown in Table 5. We train IMoJIE and OpenIE6 (initialized with mBERT) on AACTRANS+CLP and Sent-Trans+CLP data. We find that they underperform 2508 SentenceExtractions George Bluth Sr., patriarch of the Bluth family, is the founder and former CEO of the Bluth Company. <s> George Bluth Sr. </s> <r> is patriarch of </r> <o> the Bluth family </o> <s> George Bluth Sr. </s> <r> is </r> <o> the founder and former CEO of the Bluth Company </o> <s> George Bluth Sr. </s> <r> is </r> <o> patriarch of the Bluth family </o> TeluguEnglishExtraction  Sharon's longtime rival Benjamin Netanyahu was elected as leader of Likud <s>  </s> <o>  </o> <r>  </r> HindiEnglishExtraction               (cid:607)   John Lambert put forward a new constitution known as the Instrument of Government <s>   (cid:607) </s> <o>       </o> <r>    </r> Table 5: Sentence and OpenIE predictions of GEN2OIE in English, Telugu and Hindi. GEN2OIE and Multi 2 OIE. Compared to the two-stage models, both IMoJIE and OpenIE6 generate all the extractions autoregressively, which makes them more susceptible to noise in the automatically generated training data. We additionally compare with Faruqui (2015), where the test sentence is translated into English, extractions are generated using OpenIE6 and they are projected back onto the test sentence. We find that the system results in poor performance due to lack of language-specific training. We observe that all systems have low performance on Chinese. We attribute this to the various artifacts present in the SAOKE test set, that include special relations such DESC , TIME , ISA , etc. Since these extractions cannot be generated in our pipeline, we observe performance of only 33.2% F1 and 15.8% AUC with our best model, when compared to training GEN2OIE with SAOKE training data, which gives 52.5% F1 and 32% AUC. We additionally train the GEN2OIE model using mT5 on AACTRANS data for all five languages (GEN2OIE-mT5 in Table 4) and find improvements of (2.1%, 3.5%, 0.8%) F1 over the mBART models used for ES, ZH and HI. In order to measure the inconsistency of the generated extractions with respect to the sentence, we compute the fraction of words that occur in the extraction but are absent in the sentence. In Table 7, we find that across languages, the fraction is lower for training examples generated through the consistent translation methodology (AACTRANS+CLP) when compared against independent translations (SentExtTrans+CLP). This indicates that AAC-TRANS+CLP indeed achieves better consistency. In order to analyze the reasons for improvement in CaRB performance, we compute the fraction of words that are present in model predictions but absent in the gold extractions of the test set (denoted by AG Absent in Gold). In Table 8, we see that GEN2OIE trained on AACTRANS+CLP achieves lower values than the same model trained on Sen-tExtTrans+CLP data and this correlates with the increased CaRB performance. This shows that the model generates words closer to gold extractions (and hence closer to input sentence), which contributes to higher performance. We choose three representative languages to conduct the ablation study  Spanish, Chinese, and Hindi. Portuguese and Telugu belong to the same language family as Spanish and Hindi, respectively. In Table 6, we show the results of individually removing components from the GEN2OIE trained on AACTRANS+CLP data. In AACTRANS w/o Sentence Consistency, we use regular translation of sentence while using consistent translation of extraction. This leads to a drop of (1.9, 0.2, 0.9)% in F1 for the three languages, and shows the importance of using consistent translation on both the sentence and extraction. In GEN2OIE w/o Relation Ordering, we train Stage-1 GEN2OIE with randomly shuffled relations. This reduces the performance as our model uses auto-regressive training which benefits from following a fixed order, which we choose as the order of occurrence of the relations in the sentence. In GEN2OIE w/o Relation Coverage, we find that performance decreases in Spanish and Chinese by 5.3% and 5.9% in F1, respectively, but remains the same in Hindi, possibly due to the smaller number of examples in the test set. Error Analysis : We find that the AAC-TRANS+CLP suffers from: 1) missing or 2) wrong word alignments and 3) inability to label discontinuous S, R, O phrases. We show examples of these cases in Appendix B. 9 Conclusion We develop a novel AACTRANS+CLP pipeline for consistently transferring English OpenIE examples to other languages and present a novel two-stage generative model, GEN2OIE, for training OpenIE systems in various languages. We show improvements over the existing baseline of Multi 2 OIE, with an average improvement of 7.2% in F1 and 16.1% in AUC. It is effective in five languages, which is the largest number of languages covered by a single OpenIE technique known to us. To encourage research in medium and low-resource languages, we additionally release new OpenIE evaluation examples in Hindi and Telugu. Keshav is supported by TCS Research Fellowship. Mausam is supported by grants from Huawei, Google, Bloomberg and IBM, and a Jai Gupta Chair Fellowship. Soumen is partly supported by a Jagadish Bose Fellowship and an AI Horizons Network grant from IBM. We thank IIT Delhi HPC facility and TFRC program for compute resources.
abstain objective result objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain abstain objective method objective objective objective objective objective other abstain other other other other other other abstain other other other other other other other method method abstain other other abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain other abstain abstain abstain result result abstain result result abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result result abstain result abstain abstain other other other other