We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average. 1 1 Introduction Multilingual embedding spaces have been demonstrated to be a promising means for enabling crosslingual transfer in many natural language processing tasks (e.g. Ammar et al. (2016); Lample et al. (2018)). Similar to how universal part-of-speech tags enabled parsing transfer across languages (Petrov et al., 2012), multilingual word embeddings further improve transfer capacity by enriching models with lexical information. Since this lexical representation is learned in an unsupervised fashion and thus can leverage large amounts of raw data, it can capture a more nuanced representation of meaning than unlexical-ized transfer. Naturally, this enrichment is transEqual contribution 1 Code and models: https://github.com/ TalSchuster/CrossLingualELMo . in low-resource scenarios (Guo et al., 2015). In this paper, we are moving further along this line and exploring the use of contextual word embeddings for multilingual transfer. By dynamically linking words to their various contexts, these embeddings provide a richer semantic and syntactic representation than traditional context-independent word embeddings (Pe-ters et al., 2018). A straightforward way to utilize this richer representation is to directly apply existing transfer algorithms on the contextual embeddings instead of their static counterparts. In this case, however, each token pair is represented by many different vectors corresponding to its spe-cific context. Even when supervision is available in the form of a dictionary, it is still unclear how to utilize this information for multiple contextual embeddings that correspond to a word translation pair. In this paper, we propose a simple but effective mechanism for constructing a multilingual space of contextual embeddings. Instead of learning the alignment in the original, complex contextual space, we drive the mapping process using context-independent embedding anchors. We obtain these anchors by factorizing the contextual embedding space into context-independent and context-dependent parts. Operating at the anchor level not only compresses the space, but also enables us to utilize a word-level bilingual dictionary as a source of supervision, if available. Once the anchor-level alignment is learned, it can be readily applied to map the original spaces with contextual embeddings. Clearly, the value of word embeddings depends on their quality, which is determined by the amount of raw data available for their training (Jiang et al., 2018). We are interested in expanding the above approach to the truly low-resource scenario, where a language not only lacks annotations, but also has limited amounts of raw data. In this case, we can also rely on a data rich language to stabilize monolingual embeddings of the resource-limited language. As above, context-independent anchors are informing this process. Specifically, we introduce an alignment component to the loss function of the language model, pushing the anchors to be closer in the joint space. While this augmentation is performed on the static anchors, the benefit extends to the contextual embeddings space in which we operate. We evaluate our aligned contextual embeddings on the task of zero-shot cross-lingual dependency parsing. Our model consistently outperforms previous transfer methods, yielding absolute improvement of 6.8 LAS points over the prior state-of-the-art (Ammar et al., 2016). We also perform comprehensive studies of simplified variants of our model. Even without POS tag labeling or a dictionary, our model performs on par with context-independent models that do use such information. Our results also demonstrate the benefits of this approach for few-shot learning, i.e. processing languages with limited data. Specifically, on the Kazakh tree-bank from the recent CoNLL 2018 shared task with only 38 trees for training, the model yields 5 LAS points gain over the top result (Smith et al., 2018a). Multilingual Embeddings The topic of crosslingual embedding alignment is an active area of research (Mikolov et al., 2013; Xing et al., 2015; Dinu and Baroni, 2014; Lazaridou et al., 2015; Zhang et al., 2017). Our work most closely relates to MUSE (Conneau et al., 2018a), which constructs a multilingual space by aligning monolingual embedding spaces. When a bilingual dictionary is provided, their approach is similar to those of (Smith et al., 2017; Artetxe et al., 2017). MUSE extends these methods to the unsupervised case by constructing a synthetic dictionary. The resulting alignment achieves strong performance in a range of NLP tasks, from sequence labeling (Lin et al., 2018) to natural language inference (Conneau et al., 2018b) and machine translation (Lample et al., 2018; Qi et al., 2018). Recent work further improves the performance on both the supervised (Joulin et al., 2018) and unsupervised (Grave et al., 2018b; Alvarez-Melis and Jaakkola, 2018; Hoshen and Wolf, 2018) settings for context-independent embeddings. While MUSE operates over token based embeddings, we are interested in aligning contextual embeddings, which have shown their benefits in several monolingual applications (Peters et al., 2018; McCann et al., 2017; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018). However, this expansion introduces new challenges which we address in this paper. In a concurrent study, Aldarmaki and Diab (2019) introduced an alignment that is based only on word pairs in the same context, using parallel sentences. Our method achieves better word translations without relying on such supervision. Our work also relates to prior approaches that utilize bilingual dictionaries to improve embeddings that were trained on small datasets. For instance, Xiao and Guo (2014) represent word pairs as a mutual vector, while Adams et al. (2017) jointly train cross-lingual word embeddings by replacing the predicted word with its translation. To utilize a dictionary in the contextualized case, we include a soft constraint that pushes those translations to be similar in their context-independent representation. A similar style of regularization was shown to be effective for cross-domain transfer of word embeddings (Yang et al., 2017). Multilingual Parsing In early work on multilingual parsing, transfer was commonly implemented using delexicalized representation such as part-of-speech tags (McDonald et al., 2011; Petrov et al., 2012; Naseem et al., 2012; Tiedemann, 2015). Another approach for cross-lingual parsing includes annotation projection and treebank translation (Xiao and Guo, 2015; Wang and Eisner, 2016; Tiedemann, 2017), which mostly require some source of supervision. Advancements in multilingual word representations opened a possibility of lexicalized transfer. Some of these approaches start by aligning monolingual embedding spaces (Zhang and Barzilay, 2015; Guo et al., 2015, 2016; Ammar et al., 2016), and using resulting word embeddings as word representations instead of universal tags. Other approaches are learning customized multilingual syntactic embeddings bootstrapping from universal POS tags (Duong et al., 2015). While some models also learn a language embedding (Ammar et al., 2016; de Lhoneux et al., 2018), it is unfeasible in a zero-shot scenario. In all of the above cases, token-level embeddings are used. Inspired by strong results of using contextualized embeddings in monolingual parsing (Che et al., 2018; Wang et al., 2018; Clark et al., 2018), we aim to utilize them in the multilingual transfer case. Our results demonstrate that richer representation of lexical space does lead to significant performance gains. In this section we describe several approaches for aligning context-dependent embeddings from a source language s to a target language t . We address multiple scenarios, where different amounts of supervision and data are present. Our approach is motivated by interesting properties of context-dependent embeddings, which we discuss later. We begin with some notations:  Context Dependent Embeddings: Given a context c and a token i , we denote the embedding of i in the context c by e i,c . We use e i,  to denote the point cloud of all contextual embeddings for token i . Embedding Anchor: Given a token i we denote the anchor of its context dependent embeddings by  e i , where:  e i = E c (cid:2) e i,c (cid:3) . (1) In practice, we calculate the average over a subset of the available unlabeled data. Shift From Mean: For any embedding e i,c we can therefore define the shift  e i,c from the average via: e i,c =  e i +  e i,c . Formally, our alignment is always of the following form: e s  t i,c = W s  t e si,c . A given token i can generate multiple vectors e i,c , each corresponding to a different context c . A key question is how the point cloud e i,  is distributed. In what follows we explore this structure, and reach several conclusions that will motivate our alignment approach. The following experiments are performed on ELMo (Peters et al., 2018). Point Clouds are Well Separated A cloud e i,  corresponds to occurrences of the word i in different contexts. Intuitively, we would expect its points to be closer to each other than to points from e j,  for a different word j . Indeed, when measuring similarity between points e i,c and their anchor  e i , we find that these are much more similar than anchors of different words  e i and  e j (see Table 1). This observation supports our hypothesis that anchor-driven alignment can guide the construction of the alignment for the contextual space. A visualized example of the contextualized representations of four words is given in Figure 1, demonstrating the appropriateness of their anchors. Still, as previous studies have shown, and Figure 2: Contextual embeddings for the English word bear and its two possible translations in Spanish  oso (animal) in blue and tener (to have) in red. Homonym Point Clouds are Multi-Modal When a word i has multiple distinct senses, we might expect the embeddings for i to reflect this by separating into multiple distinct clouds, one for each meaning. Figure 2 demonstrates that this indeed happens for the English word bear. Furthermore, it can be seen that after alignment (Sec-tion 3.3) with Spanish, the distinct point clouds are aligned with their corresponding distinct words in Spanish. See App. D for another example. We examined the shift from mean for a list of 250 English homonyms from Wikipedia. 2 As Table 1 shows, the shift of these words is indeed slightly higher than it is for other words. However, they still remain relatively close to their per-token anchor. Therefore, these anchors can still serve as a good approximation for learning alignments. We begin by briefly reviewing previous approaches for aligning context-independent embeddings, as they are generalized in this work to the contextual case. We denote the embedding of a word i by e i . At first, assume we are given word pairs { ( e si , e ti ) } from a source language s and a 2 https://en.wikipedia.org/wiki/List_ of_true_homonyms target language t , and we look for a mapping between those. Mikolov et al. (2013) proposed to learn a linear transformation whereby e ti is approximated via W e si , for a learned matrix W . We focus on methods that follow this linear alignment. The alignment matrix is found by solving: W s  t = argmin W  O d ( R ) n (cid:88) i =1 (cid:13)(cid:13)(cid:13) W e si  e ti (cid:13)(cid:13)(cid:13) 2 , (4) where O d ( R ) is the space of orthogonal matrices. This constraint was proposed by Xing et al. (2015) in order to preserve inter-lingual relations. Under this constraint, Eq. 4 is an instance of the orthogonal Procrustes problem, which has a closed-form solution W s  t = UVT . The columns of U and V are the left and right singular vectors of the multiplication of the source and (transposed) target embedding matrices. For the unsupervised case (i.e. when a dictionary is absent), Conneau et al. (2018a) (MUSE) suggested to learn the alignment via adversarial training, such that a discriminator is trained to distinguish between target and aligned source embeddings. Thereafter, a refinement procedure is applied iteratively as follows. First, a dictionary is built dynamically using the current alignment such that only words with high confidence are considered. Using the dictionary, the alignment matrix is re-calculated as in the supervised case. We next turn our attention to the main task of this paper, which is aligning context-dependent embeddings. We now describe our generalization of the methods described in Section 3.2 for this case. The first two methods are based only on anchors while the third one uses the contextual vectors themselves. Altogether, we suggest three alignment procedures, one aimed for the supervised and two for the unsupervised cases. Supervised Anchored Alignment As a first step, we are assuming access to a dictionary for the source and target domains. For each source word i denote by D ( i ) the corresponding word in the target language. 3 In the context-dependent case, Eq. 3 In practice, we may have multiple target words for a single source word, and the extension is straight-forward. words. However, this challenge can be addressed by aligning the vectors  e i for which we do have one per word. This is motivated by our observations in Section 3.1 that context-dependent embeddings are well clustered around their centers. we solve Eq. 4 with token anchors as inputs. We emphasize that by constraining W s  t to be orthogonal, we also preserve relations between  e i,c and  e i,c (cid:48) that represent the contextual information. Unsupervised Anchored Alignment In this setting, no dictionary is present. As in the supervised case, we can naturally extend a context-independent alignment procedure to the contextual space by leveraging the anchor space  e i . This can be done using the adversarial MUSE framework proposed by Conneau et al. (2018a) and described at the end of Section 3.2. Unsupervised Context-based Alignment Alternatively, the alignment could be learned directly on the contextual space. To this end, we follow again the adversarial algorithm of MUSE, but for each word we use multiple embeddings induced by different contexts, rather than the word anchor. This context-based alignment presents opportunities but also introduces certain challenges. On the one hand, it allows to directly handle homonyms during the training process. However, empirically we found that training in this setting is less stable than unsupervised anchored alignments. Refinement As a final step, for both of the unsupervised methods, we perform the refinement procedure that is incorporated in MUSE (end of Section 3.2). In order to synthesize a dictionary, we use distance in the anchor space. Thus far we assumed that embeddings for both source and target languages are pretrained separately. Afterwards, the source is mapped to the target in a second step via a learned mapping. However, this approach may not work well when raw data for the source languages is scarce, resulting in deficient embeddings. In what follows, we show how to address this problem when a dictionary is available. We focus on embeddings that are learned using a language model objective but this can be easily generalized to other objectives as well. Our key idea is to constrain the embeddings across languages such that word translations will be close to each other in the embedding space. This can serve as a regularizer for the resource-limited language model. In this case, the anchors are the model representations prior to its context-aware components (e.g., the inputs to ELMo's LSTM). Denote the anchor for word i in language s by v si . Now, assume we have trained a model for the target language and similarly have embeddings v ti . We propose to train the source model with an added regularization term as follows:  anchor  (cid:88) i (cid:107) v si  v tD ( i ) (cid:107) 22 , (5) where  anchor is a hyperparamter. This regularization has two positive effects. First, it reduces overfitting by reducing the effective number of parameters the model fits (e.g., if the regularizer has large coefficient, these parameters are essentially fixed). Second, it provides a certain level of alignment between the source and target language since they are encouraged to use similar anchors. Now that we presented our method for aligning contextual embeddings, we turn to evaluate it on the task of cross-lingual dependency parsing. We first describe our baseline model, and then show how our alignment can easily be incorporated into this architecture to obtain a multilingual parser. Baseline Parser Most previous cross-lingual dependency parsing models used transition-based models (Ammar et al., 2016; Guo et al., 2016). We follow Che et al. (2018); Wang et al. (2018); Clark et al. (2018) and use a first-order graph-based model. Specifically, we adopt the neural edge-scoring architecture from Dozat and Manning (2017); Dozat et al. (2017), which is based on Kiperwasser and Goldberg (2016). We now briefly review this architecture. Given a sentence s , let e i and p i be its word and POS-tag embeddings. These are concatenated and fed into a Bi-LSTM to produce token-level contextual representations r i . Four Multi-Layer Perceptrons are applied on these vectors, resulting in new representations h arc  dep i , h arc  head i , h rel  dep i and h rel  head i for each word i . s arcij = (cid:16) h arc  head i (cid:17) T (cid:16) U arc h arc  dep j + b arc (cid:17) . (6) Additionally, the score for predicting the dependency label r for an edge ( i, j ) is defined as s rel ( i,j ) ,r = (cid:16) h rel  head i (cid:17) TU relr h rel  dep j + (cid:16) u rel  head r (cid:17) T h rel  head i + (cid:16) u rel  dep r (cid:17) T h rel  dep j + b r . (7) At test time, MST is calculated to ensure valid outputs. Multilingual Parsing with Alignment We now extend this model, in order to effectively use it for transfer learning. First, we include contextualized word embeddings by replacing the static embeddings with a pre-trained ELMo (Peters et al., 2018) model (instead of e i ). Second, we share all model parameters across languages and use the contextual word embeddings after they are aligned to a joint space J . Formally, if s is a sentence of language (cid:96) , contextual word embeddings are obtained via: e (cid:96)  J i,s = W (cid:96)  J e i,s , (8) where W (cid:96)  J is the alignment matrix from language (cid:96) to the joint space. 4 This alignment is learned apriori and kept fixed during parser training. This setup is applicable for both single and multiple training languages. For the tested language, training data could be available, sometimes limited (few-shot), or absent (zero-shot). The alignment methods are described in detail in Section 3. In their paper, Peters et al. (2018) suggest to output a linear combination over the representations of each layer of ELMo, learning these weights jointly with a downstream task. Our alignment is learned separately for each layer. Therefore, we keep the weights of the combination fixed during the training to ensure that the parser's inputs are from the joint cross-lingual space. Alternatively, one can share the weights of the combination between the languages and learn them. 4 We use the space of the training language as our joint space and align the tested language to it. In the multi-source scenario, we align all embeddings to English. Contextual Embeddings We use the ELMo model (Peters et al., 2018) with its default parameters to generate embeddings of dimension 1024 for all languages. For each language, training data comprises Wikipedia dumps 5 that were tokenized using UDpipe (Straka and Strakova, 2017). We randomly shuffle the sentences and, following the setting of ELMO, use 95% of them for training and 5% for evaluation. Alignment We utilize the MUSE framework 6 (Conneau et al., 2018a) and the dictionary tables provided by them. The  e i (anchor) vectors for the alignment are generated by computing the average of representations on the evaluation set (except for the limited unlabeled data case). To evaluate our alignment, we use the anchors to produce word translations. For all experiments we use the 50k most common words in each language. Dependency Parsing We used the biaffine parser implemented in AllenNLP (Gardner et al., 2018), refactored to handle our modifications as described in Section 4. 7 The parser is trained on trees from a single or multiple languages, as described in each setting (Section 6). For the multiple case, we randomly alternate between the available languages, i.e. at each iteration we randomly choose one language and sample a corresponding batch. Dropout (Srivastava et al., 2014) is applied on ELMo representations, Bi-LSTM representations and outputs of MLP layers. We also apply early stopping, where validation accuracy is measured as average LAS score on the development set across all training languages. The parser hyperparameters are the same as Dozat et al. (2017) except we reduce the POS tag embedding size from 100 to 50 and increase the head/dependent MLP dimension from 400 to 500. All hyperparameter values used are listed in App. C. From experiments on the English tree-bank, we found that using the outputs of the first LSTM layer is as good as learning a combination. 8 This 5 https://lindat.mff.cuni.cz/ repository/xmlui/handle/11234/1-1989 6 https://github.com/facebookresearch/ MUSE/ 7 https://github.com/TalSchuster/ allennlp-MultiLang 8 This was concurrently justified by Liu et al. (2019), showing that the first layer alone can perform better than a mixture. Therefore, we fix the weights over ELMo layers to [0 , 1 , 0] , i.e. using only representations from the first LSTM layer. Evaluation Scenarios for Dependency Parsing For a fair comparison, we use the same setting as used by previous models for each scenario. Our main model (which we refer to as OURS ) is using a SUPERVISEDANCHOREDALIGNMENT (Section 3.3) to align the multilingual pretrained ELMo embeddings which are used by the parser. We compare against several variants of our model:  ALIGNED FASTTEXT : instead of ELMo, we use FASTTEXT pretrained embeddings (Grave et al., 2018a), aligned to English using MUSE. ALIGNED  e : instead of contextualized embeddings, we use the anchors themselves as fixed embeddings, aligned to English. Alignment As mentioned above, we use outputs of the first LSTM layer of ELMo in our parsing experiments. Therefore, we present the alignment accuracy for those in Table 2, summarizing the precision@5 word-translation from 6 languages to English. Results for the other layers are presented in App. A. As expected, supervised alignments outperform unsupervised ones by a large margin. Between the two unsupervised methods, the context-based alignment achieved significantly better results for Spanish and Portuguese but failed to converge for Swedish. In both cases, the value of anchors in the REFINE step is clear, substantially improving the precision for all languages. Zero-Shot Parsing, Multiple Source Languages Table 3 summarizes the results for our zero-shot, multi-source experiments on six languages from Google universal dependency treebank version 2.0. 9 For each tested language, the parser was trained on all treebanks in the five other languages and English. We align each of the six languages to English. We compare our model to the performance of previous methods in the same setting (re-ferred to as L t  L s =  in Ammar et al. (2016)). The results show that our multilingual parser outperforms all previous parsers with a large margin of 6.8 LAS points. Even with an unsupervised alignment, our model consistently improves over previous models. To make a fair comparison to previous models, we also use gold POS tags as inputs to our parser. However, for low-resource languages, we might not have access to such labels. Even without the use of POS tags at all, in five out of six languages the score is still higher than previous methods that do consider such annotations. An exception is the Portuguese language where it leads to a drop of 8.8 LAS points. While in the single language setting this good performance can be explained by the knowledge captured in the character level, contextual embeddings (Smith et al., 2018b; Belinkov 9 https://github.com/ryanmcd/ uni-dep-tb/ et al., 2017), the results suggest that this knowledge transfers across languages. In order to assess the value of contextual embeddings, we also evaluate our model using non-contextual embeddings produced by FASTTEXT (Bojanowski et al., 2017). While these improve over previous works, our context-aware model outperforms them for all six languages in UAS score and for 5 out of 6 languages in LAS score, obtaining an average higher by 3 points. To further examine the impact of introducing context, we run our model with precomputed anchors (  e ). Unlike FASTTEXT embeddings of size 300, these anchors share the same dimension with contextual embeddings but lack the contextual information. Indeed, the context-aware model is consistently better. Few-Shot Parsing, Small Treebanks In this scenario, we assume a very small tree-bank for the tested language and no POS tags. We use the Kazakh tree-bank from CoNLL 2018 shared task (Zeman et al., 2018). The training set consists of only 38 trees and no development set is provided. Segmentation and tokenization are applied using UDPipe. Similar to Rosa and Marecek (2018); Smith et al. (2018a), we utilize the available training data in Turkish as it is a related language. To align contextual embeddings, we use a dictionary generated and provided by Rosa and Marecek (2018) and compute an alignment from Kazakh to Turkish. The dictionary was obtained using FastAlign (Dyer et al., 2013) on the Open-Subtitles2018 (Lison and Tiedemann, 2016) parallel sentences dataset from OPUS (Tiedemann, 2012). 10 Table 5 summarizes the results, showing that our algorithm outperforms the best model from the shared task by 5 . 05 LAS points and improves by 10 https://github.com/CoNLL-UD-2018/ CUNI-x-ling over 10 points over a FASTTEXT baseline. Zero-Shot Parsing, Limited Unlabeled Data To evaluate our anchored language model (Sec-tion 3.4), we simulate a low resource scenario by extracting only 10k random sentences out of the Spanish unlabeled data. We also extract 50k sentences for LM evaluation but perform all computations, such as anchor extraction, on the 10k training data. For a dictionary, we used the 5k training table from Conneau et al. (2018a). 11 Another table of size 1,500 was used to evaluate the alignment. In this scenario, we assume a single training language (English) and no usage of POS tags nor any labeled data for the tested language. Table 4 shows the results. Reducing the amount of unlabeled data drastically decreases the precision by around 20 points. The regularization introduced in our anchored LM significantly improves the validation perplexity, leading to a gain of 7 UAS points and 9 LAS points. We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings of different languages, pretrained in an unsupervised fashion. At the core of our methods, we suggest to use anchors for tokens, reducing this problem to context-independent alignment. Our methods are compatible both for cases where a dictionary is present and absent, as well as for low-resource languages. The acquired alignment can be used to improve cross-lingual transfer learning, gaining from the contextual nature of the embeddings. We show that these methods lead to good word translation results, and improve significantly upon state-of-the-art zero-shot and few-shot crosslingual dependency parsing models. In addition, our analysis reveals interesting properties of the context-aware embeddings generated by the ELMo model. Those findings are another step towards understanding the nature of contextual word embeddings. We thank the MIT NLP group and the reviewers for their helpful discussion and comments. The first and third authors were supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work was also supported in part by the US-Israel Binational Science Foundation (BSF, Grant No. 2012330), and by the Yandex Initiative in Machine Learning.
objective abstain method abstain objective result abstain abstain abstain other abstain objective abstain abstain abstain abstain objective method result method abstain abstain method method abstain abstain objective method result method method objective abstain other method other other other other abstain objective other abstain abstain other method other other other other other other other other abstain objective abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method abstain method method method abstain method abstain method method abstain abstain other abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result objective method abstain abstain result result abstain other other other other other