Cross-lingual named entity recognition task is one of the critical problems for evaluating the potential transfer learning techniques on low resource languages. Knowledge distillation using pre-trained multilingual language models between source and target languages have shown their superiority in transfer. However, existing cross-lingual distillation models merely consider the potential transferability between two identical single tasks across both domains. Other possible auxiliary tasks to improve the learning performance have not been fully investigated. In this study, based on the knowledge distillation framework and multitask learning, we introduce the similarity metric model as an auxiliary task to improve the cross-lingual NER performance on the target domain. Specifically, an entity recognizer and a similarity evaluator are first trained in parallel as two teachers from the source domain. Then, two tasks in the student model are supervised by these teachers simultaneously. Empirical studies on the three datasets across 7 different languages confirm the effectiveness of the proposed model. Named entity recognition, NER in short, refers to identifying entity types, i.e. location, person, organization, etc., in a given sentence. The exploiting of deep neural networks, such as Bi-LSTM-CRF (Lample et al., 2016), Bi-LSTM-CNN (Chiu and Nichols, 2016) makes this task achieve significant performances. However, since deep neural networks highly rely on a large amount of labelled training data, the annotation acquiring process is expensive and time consuming. This situation is more severe for zero-resource languages. With the help of transfer learning (Ruder et al., 2019) and multilingual BERT (short as mBERT) (Devlin et al.,  Corresponding author NER stu TSL MTMTNER tea {X,P} tgt NER {X,Y} src NER tea Directly {X} tgt {X} tgt NER stu training training s o f t l a b e li n g SIM tea {X,S} tgt training soft labeling similarityscore {X,P} tgt collaboration Figure 1: Comparison between previous cross-lingual NER models. Directly : direct model transfer; TSL : teacher-student learning model; MTMT : proposed multiple-task and multiple-teacher Model. NER / NER tea : learned NER model for source language; NER stu : learned NER model for target language; SIM tea learned similarity model for source language; { X, Y } src : labeled data in source language; { X } tgt : unlabeled data in target language; { X, P } tgt : labeled data in target language with probability; { X, S } tgt : labeled data in target language with entity similarity score. 2019), it is possible to transfer the annotated training samples or trained models from a rich-resource domain to a zero-resource domain. Many studies have been done to solve this cross-lingual NER problem. Existing models can be separated into three categories, shared feature space based, translation based and knowledge distillation based. Shared feature space based models exploit language-independent features, which lacks the domain-specific features for the target language (Tsai et al., 2016; Wu and Dredze, 2019; Keung et al., 2019). Translation based models generate pseudo labeled target language data to train the cross-lingual NER model, but the noise from translation process restrains its performance. (Mayhew et al., 2017; Xie et al., 2018; Wu et al., 2020b). Knowledge distillation based models train a student model using soft labels of the target language (Wu et al., 2020a,b; Chen et al., 2021; Liang et al., 170 2021). Although the above-mentioned models solve the cross-lingual NER problem to some extent, the auxiliary tasks, as in multi-task learning, have not been studied in this problem. Due to the distributed representation of natural languages, the relatedness among the embedding of target languages, which is measured by the similarity, can be utilized to further boost the learned encoder and improve the final NER performance on target languages. Here we give a concrete example to illustrate the importance of similarity between every two tokens under the situation when only the English data is labeled. Given a Spanish sentence  Arvalo (Avila), 23 may (EFE). , the token  Arvalo  is recognized as ORG type using the learned model from the English domain. In the meantime, the token  Arvalo  has high similarity scores with the Spanish tokens  Viena  from sentence  Viena, 23 may (EFE). ", and  Madrid  from sentence  Madrid, 23 may (EFE). . Also, the tokens  Viena  and  Madrid  are recognized correctly as LOC type using the same English model mentioned above. Then  Arvalo  can be recognized correctly as LOC type under the supervisory signal using the similarity between  Viena  and  Madrid . To leverage the similarity between the tokens of the source languages, we design an multiple-task and multiple-teacher model (short as MTMT, as shown in Figure 1), which helps the NER learning process on the target languages. Specifically, we first introduce the knowledge distillation to build entity recognizer and similarity evaluator teachers in the source language and transfer the learned patterns to the student in the target language. In the student model, we then borrow the idea of multitask learning to incorporate a similarity evaluation task as an auxiliary task into the entity recognition classifier. During the student learning process, we input unlabelled samples from the target languages into the entity recognizer and evaluator, and take output pesudo labels as supervisory signals for these two tasks in the student model. Note that a weighting strategy is also provide therein to take into consideration of the reliability of the teachers. We validate the model performance on the three commonly-used datasets across 7 languages and the experimental results show the superiority of our presented MTMT model. distillation framework for cross-lingual named entity recognition and develop a teaching and learning procedure under this framework. We present a novel multiple-task and multiple-teacher model that introduces an entity similarity evaluator to boost the performance of student recognizer on target languages. We conduct extensive experiments on 7 languages compared with state-of-the-art baselines and the results confirm the effectiveness of the presented model. Our approach is closely related to the existing works on cross-lingual NER, knowledge distillation, and siamese network. Cross-Lingual NER aims to extract entities from a target language but assumes only source language is annotated. The existing models can be categorized to a) Shared feature space based models, b) Translation based models, c) Knowledge distillation based models. Shared feature space based models generally train a language-independent encoder using source and target language data (Tsai et al., 2016). Recently, the pre-trained multilingual language model is effective to address the challenge (Devlin et al., 2019). Moreover, some research introduces new components on top of the mBERT by directly transferring the model learned from the labeled source language to that of target languages (Keung et al., 2019). The performance is still weak due to the lack of annotations of target languages. Translation based models generally generate pseudo-labeled target data to alleviate target data scarcity. For example, (Wu et al., 2020b; Zhang et al., 2021) gain an improvement by translating the labeled source language to the target language word-by-word. Our model achieves considerable improvement by learning entity similarity in target language data without translation. Knowledge distillation based models include a teacher model and a student model (Wu et al., 2020c). The teacher model is trained on the labeled source language. The student model learns from the soft label predicted by the teacher model on unlabeled target language data. Therefore, the student model can capture the extra knowledge about target languages. In our work, the student model not only learns the recognizer teacher knowledge, but also 171 learns the entity similarity knowledge inspired by multi-task learning. Siamese Network is originally introduced by (Bromley et al., 1994) to treat signature verifica-tion as a matching problem. It has been successfully applied to transfer learning such as one-shot image recognition (Koch et al., 2015), text similarity (Neculoiu et al., 2016). However, there is a dilemma to adapt the siamese network to token-level recognition tasks such as NER. Siamese network assumes the input is a pair, and the output is a similarity score. To handle this issue, we reconstruct the data to pair format. To the best of our knowledge, we are the first to learn the entity similarity by siamese network. In this section, we introduce our framework and its detailed implementation. Our framework is consist of two models: teacher training model learned from the source language and teacher-student distillation learning model learned from the target language. In the teacher training model, there are two sub-models, i.e. an entity recognizer teacher and a similarity evaluator teacher. These two models are two parallel tasks, wherein the entity recognition teacher focuses on identifying the named entities and the similarity evaluator teacher is to decide if two tokens are in the same type. We then present a teacher-student distillation learning model to learn from the two learned teacher models simultaneously. We note that, in this learning process, such a knowledge distillation makes the student model combine the advantages of both source language patterns of entity recognition and entity similarity evaluation. During the learning process, the samples from the target language are fed into the teacher model and the outputs are taken as the supervisory signal for two tasks in the student model. To guarantee the student learning performance, we assign weights for each supervisory signal correspond to the output confidence of teacher sub-models. We argue that the student entity recognition task and the student entity similarity evaluation task improve the representation learning of the student encoder in the siamese structure. Following standard practice, we formulate cross-lingual NER as a sequence labeling task. Given a EvaluatorTeacher E n c o d e r C o s E n c o d e r Labeled Source-Language Pairwise Data L i n ea r E n c o d e r RecognizerTeacher Labeled SourceLanguage Data reconstruct training training CELoss BCELoss Figure 2: The training process of teacher models. sentence x = { x i } Li =1 with L tokens, a NER model produces a sequence of labels y = { y i } Li =1 , where x i is the i -th token and y i is the corresponding label of x i . In the source language, we denote the labeled training data as D Strain = { ( x , y ) } and test data as D Stest . In the target language, we denote the unlabeled train data as D Ttrain = { x } and the test data as D Ttest . Formally, our goal is to train a model with D Strain and D Ttrain to perform well on D Ttest . Here we first consider the training of two teacher models. For every two tokens, we define Entity Similarity Metric as a score which is the probability that two tokens belong to the same entity type. We aim to find entity similarity to help the cross-lingual NER model in the target language. It is a non-trivial task since we lack golden labels to help us distinguish target named entities. To address this challenge, we propose a binary classifier called similarity evaluator to leverage the labeled source language data for similarity prediction. Our similarity evaluator model, inspired by siamese network (Koch et al., 2015), are able to acquires more powerful features via capturing the invari-ances to transformation in the input space. Figure 2 illustrated the two teacher models training. The following subsections will illustrate the two teacher models sequentially. Since the cross-lingual NER task, we unitize multilingual mBERT (Wu and Dredze, 2019) as basic sequence feature extractor backbone to derive the sequence embedding representation throughout this paper. And a linear classifier with softmax upon the pre-trained mBERT output. The model network 172 structure could be formulated as, h = mBERT ( x )  y i = softmax ( W h i + b ) where h = { h i } Li =1 and h i denotes the output of the pre-trained mBERT that corresponds to the input token x i . y i denotes the predicted probability distribution for x i . W and b are trainable parameters. For some sentence sample ( x , y )  D Strain and an entity token query index i , the loss function is, LER ( x , y , i ) = LCE ( y i ,  y i ) We train this entity recognition teacher model on the source lingual training corpus D Strain = { ( x , y ) } directly. To leverage the entity similarity to boost the unsupervised cross-lingual NER performance, we will present our entity pairs construction method and the siamese network model in the following. Entity Similarity Pairs Construction According to entity labels, we randomly select sentences pair < x , x (cid:48) > with their some token pair < x i , x (cid:48) j > and associated labels < y i , y (cid:48) j > in D Strain , to form the siamese supervision training dataset, DS  siam train = { ( x , x (cid:48) , i, j, t ) } where the target t = 1 indicates y i = y (cid:48) j , and 0 otherwise. And the testing entity pairs DS  siam test is constructed likewisely. Siamese Entity Similarity Network Our similarity backbone model is a siamese neural network with mBERT as feature extraction layer. Wherein h and h (cid:48) represent latent sequences encoding features derived by the two symmetric twins with respect to input sentence x and x (cid:48) respectively. The inter-entities similarity is measured on the hidden representations h i and h (cid:48) j of the tokens queried by the entity indices < i, j > on the sequences representations. The cosine function operator is added to compute on the entity token latent vectors' distance, s to measure the similarity between each siamese twin, which is fed into a single sigmoid output unit for target  t estimation. More precisely, for a specific entity pair ( x , x (cid:48) , i, j, t )  DS  siam train , the siamese network could be formulated as, h = mBERT ( x ) , h (cid:48) = mBERT ( x (cid:48) )  t ( x , x (cid:48) , i, j ) =  (cos( h i , h (cid:48) j )) CELoss BCELoss Loss SimilarityScore SimilarityScore E n c o d e r C o s L i n ea r Student E n c o d e r L i n ea r E n c o d e r EvaluatorTeacher E n c o d e r C o s E n c o d e r RecognizerTeacher Teacher Inference Student Training Unlabeled Target-Language Pairwise Data L i n ea r CELoss Figure 3: Teacher-student distillation learning. where cos is the cosine similarity metric function,  is the sigmoid activation function,  t  [  (  1) ,  (1)] denotes the predicted similarity of two queried tokens pair < x i , x (cid:48) j > . Larger  t value indicates higher similarity between the two queried entities tokens. The loss function of the similarity prediction can be formulate as, LSIM ( x , x (cid:48) , i, j, t ) = LBCE ( t,  t ) . Finally, we can train the siamese entity similarity evaluator on DS  siam train , and evaluate the performance on test dataset DS  siam test . Together with entity recognizer model, this entity similarity evaluator are used as teachers in following knowledge distillation learning process, and transfer knowledge from source to target lingual corpus. In this section, we consider transferring the named entity type and similarity knowledge learned on labeled source language corpus to unlabeled target language NER task. To this end, we propose a knowledge distillation learning process to train a target language student NER model with its supervisory signals mimicked by the entity type prediction probability by the entity recognizer teacher model and entity representation similarity target by the entity siamese similarity evaluator teacher model. Based on the original unlabeled target sentence training data D Ttrain , we again construct unlabeled target-language siamese pairwise entity data DT  sim train = { ( x T , x (cid:48) T , i, j ) } , with the sentence pair < x T , x (cid:48) T > randomly sample from D Ttrain and the entity token indices pair < i, j > uniformly sampled from the sentences therein. The mBERT is also used as an encoder for the sentence siamese pair, and the entity token feature is queried from the latent sequence encoding representation. Specifically, for a sentence pair ( x T , x (cid:48) T , i, j )  DT  sim train , the student model transform them as follows, h T = mBERT ( x T )  y T i = softmax ( W h T i + b ) h (cid:48) T = mBERT ( x (cid:48) T )  y (cid:48) T j = softmax ( W h (cid:48) T j + b )  t T ( x T , x (cid:48) T , i, j ) =  (cos( h T i , h (cid:48) T j )) Then for a specific sentence pair sample in the target siamese dataset, the student loss function has three breaches, LER ( x T , y S , i ) , LER ( x (cid:48) T , y (cid:48) S , j ) , and LSIM ( x T , x (cid:48) T , i, j,  t S ) . Note that supervision information y S , y (cid:48) S , and  t S are taught by the three teacher models. Summering over all the samples in DT  sim train = { ( x T , x (cid:48) T , i, j ) } , the total student model training loss takes form, L =  (cid:88) ( x T , x (cid:48) T ,i,j ) D T  sim train (  1 LER ( x T , y S , i ) +  2 LER ( x (cid:48) T , y (cid:48) S , j ) +  LBCE ( t T ( x T , x (cid:48) T , i, j ) ,  t S )) where  1 ,  2 ,  , and  are weights in loss function which are set to make the student model learns less noisy knowledge from teachers. The weights are set as follows:  1 (  2 ) is an increasing function concerning the output of the entity recognizer teacher as shown in Figure 4. And  is set such that it is high when the output of the entity similarity teacher is close to 0 or 1 , and it is low when the output is close to 0 . 5 . indicates consistency level between the outputs from two teacher models, e.g. for two input tokens, if the output from entity similarity teacher is high, and the similarity level computed from the outputs of the entity recognizer teacher is low, then their consistency level is low. We want the student model to learn from the two teachers as follows: the higher the prediction of the entity recognizer teacher is (the further away from 0 . 5 the prediction of the entity similarity teacher is, the higher the consistency level is), the more accurate the prediction is, thus the more attention the student model pays attention to the input tokens, and vice versa. Therefore, we heuristically devises the three weights scheduling as functions of the inputs, 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (  ) = ( max ( y T i )) 2  = (2 t T ( x T , x (cid:48) T , i, j )  1) 2  = 1  |  (cos( y T i ,  y (cid:48) T j ))   t T ( x T , x (cid:48) T , i, j ) | 4 Experiment In this section, we evaluate our multiple-task and multiple-teacher model for cross-lingual NER and compare our model with a series of state-of-the-art models. We conducted experiments on three benchmark datasets: CoNLL2002 (Tjong Kim Sang, 2002), CoNLL2003 (Tjong Kim Sang and De Meulder, 2003) and WikiAnn (Pan et al., 2017). CoNLL2002 includes Spanish and Dutch, CoNLL2003 includes English and German, and WikiAnn includes English and three non-western languages: Arabic, Hindi, and Chinese. Each language is divided into a training set, a development set and a test set. All datasets were annotated with four entity types: LOC, MISC, ORG, and PER. Following (Wu and Dredze, 2019), all datasets are annotated using the BIO entity labelling scheme. To imitate the zero-resource cross lingual NER case, following (Wu and Dredze, 2019), we used English as the source language and other languages as the target language. In cross-lingual NER, the training set without entity label of the target language is also available when training the model. We trained the model with the labeled training set of the source language and evaluated the model on the test set of each target language. Table 1 and 2 shows the statistics of all datasets. We use PyTorch 1.7.1 to implement our model. All of the feature encoders mentioned in this paper use 174 Language Type Train Dev Test English-en Sentence 14,987 3,466 3,684 (CoNLL-2003) Entity 23,499 5,942 5,648 German-de Sentence 12,705 3,068 3,160 (CoNLL-2003) Entity 11,851 4,833 3,673 Spanish-es Sentence 8,323 1,915 1,517 (CoNLL-2002) Entity 18,798 4,351 3,558 Dutch-nl Sentence 15,806 2,895 5,195 (CoNLL-2002) Entity 13,344 2,616 3,941 Table 1: Statistics of CoNLL. pre-trained mBERT model (Devlin et al., 2019) in HuggingFace Transformer 1 , which has 12 Transformer blocks, 12 attention heads, and 768 hidden units. We set our hyperparameters empirically following (Wu et al., 2020c) with some modifications. We do not freeze any layers and we use the output of the last layer as our hidden feature vector. We set the batch size to be 32, maximum sequence length to be 128, dropout rate to be 0.2, and we use Adam as optimizer (Kingma and Ba, 2014). For the training of recognition teacher model and similarity teacher model, we set the learning rate to be 1e-5 and 5e-6 separately. For knowledge distillation, we use a learning rate of 1e-6 for the student models training. Note that if a word is divided into several subwords after tokenization, then only the first sub-word is considered in the loss function. Following (Tjong Kim Sang, 2002), we use the entity level F1-score as the evaluation metric. Moreover, we conduct each experiment 5 times and report the mean F1-score. Table 3 and 4 report the zero-resource cross-lingual NER results of different models on 6 target languages. Wiki (Tsai et al., 2016) introduces a language in-dependent model building on cross-lingual wikifi-cation for cross-lingual NER. WS (Ni et al., 2017) presents two weakly supervised approaches for cross-lingual NER. TMP (Jain et al., 2019) leverages machine translation to improve annotation projection approaches to cross-lingual NER. BERT-f (Wu and Dredze, 2019) applys the mBERT to cross-lingual NER. AdvCE (Keung et al., 2019) improves upon mBERT via adversarial learning for cross-lingual NER. TSL (Wu et al., 2020c) proposes a teacher-student learning model for cross-lingual NER. Unitrans (Wu et al., 2020b) unifies a data transfer and model transfer for cross-lingual NER. AdvPicker (Chen et al., 2021) proposes a adversarial discriminator for cross-lingual NER. RIKD (Liang et al., 2021) develops a reinforced iterative knowledge distillation for cross-lingual NER. TOF (Zhang et al., 2021) transfers knowledge from three aspects for cross-lingual NER. It can be seen that our model outperforms the state-of-the-arts. Specifically, compared with the remarkable RIKD, AdvPicker, and Unitrans, which also use knowledge distillation but ignore the entity similarity knowledge, our model obtains significant and consistent improvements in F1-score ranging from 0.23 for German[de] to 6.81 for Ara-bic[ar]. That demonstrates the benefits of our proposed MTMT model, compared to direct model transfer (Wu and Dredze, 2019). Note that BERT-f performs better than our model on the Chinese dataset due to their re-tokenization of the dataset. Moreover, compared with the latest model TOF, RIKD, Unitrans, our model requires much lower computational costs for both translation and iterative knowledge distillation, meanwhile reaching superior performance. For a fair comparison, we compare our model against the version of TOF w/o continual learning (Zhang et al., 40 30 20 10 0 10 20 20 10 0 10 20 30 OB-LOCI-LOCB-PERI-PERB-ORGI-ORGB-MISCI-MISC 30 20 10 0 10 20 30 20 10 0 10 20 30 OB-LOCI-LOCB-PERI-PERB-ORGI-ORGB-MISCI-MISC 30 20 10 0 10 20 30 30 20 10 0 10 20 30 OB-LOCI-LOCB-PERI-PERB-ORGI-ORGB-MISCI-MISC 2021), RIKD w/o IKD (Liang et al., 2021) and Unitrans w/o translation (Wu et al., 2020b) as reported in their paper. To demonstrate the effectiveness of our approach, we designed the following ablation studies. Table 5 presents the results. (1) MTST , which combines the multiple-teacher to single-teacher. That is, the teacher model has the same as the neural network structure of the student model. This causes a performance drop across all languages due to two single teachers cannot make a difference with the combination. (2) MTMT w/o weighting , which set the  (  ) ,  and  all to be 1 in the loss of student learning. It can be seen that the performance decrease in terms of F1-score ranges from 0.45 176 for Dutch(nl) to 0.98 for Spanish(es), which validates that weighting loss can bring more confident knowledge to the student model. (3) MTMT w/o similarity , which removes the similarity teacher model. In this case, our approach degrades into the single teacher-student learning model as in TSL (Wu et al., 2020a). Without the similarity knowledge fed into the student model, the performance drops significantly. We give a case study to show that the failed cases of baseline models can be corrected by our model. We try to bring up insights on why the proposed multiple-task and multiple-teacher model works. The proposed MTMT model can help to correct labels using the Entity Similarity defined in section 3.2. Specifically, if there is a set of tokens in which every two of them have a high Entity Similarity score, and one of the tokens is predicted to have a distinct label while other tokens have identical labels, then the one with the distinct label is predicted wrongly and is corrected by the student model to have the label of all other tokens. As shown in Table 6, in example #1, the entity recognizer teacher fails to identify  Arvalo  as B-ORG type, while the student model can correctly predict it. The reason lies in that the entity recognizer teacher predicts  Viena ( Madrid ) as B-LOC type correctly, and the similarity evaluator teacher predicts  Viena ( Madrid ) to have a high similarity score(0.7157, 0.7156) with  Arvalo . The student learns from both teachers and predict the correct label for  Arvalo . Examples #2 and #3 present the same results with different sentences. This section investigates the effect of embeddings of the two different teacher models. It can be seen that the embedding distribution of the student model is close to similarity evaluator teacher, as illustrated in Figure 5. We conjecture that the student model captures similarity knowledge from the similarity evaluator teacher, i.e. the same class of examples tend to cluster and the different class of examples tend to segregate in the embedding distribution. This validates the proposed MTMT model not only transfers cross-lingual NER knowledge from source language, but also learns the similarity knowledge of target language data. In this section, we evaluate the effectiveness of weight loss in student learning from a quantitative perspective. All of the following experiments are conducted on Spanish(es) data. For  analysis, we calculate the F1-score in different probability intervals of entity recognizer teacher, we find that the recognizer teacher tends to predict more correct in higher probability interval, as illustrated in Figure 6a. Therefore, the student model is better suited to the target language with learning fewer low-confidence misrecognitions for the target language. For  analysis, we observe that F1-score are increasing with the entity similarity score from 0.5 to both sides 0 and 1 in Figure 6b. The encoder of the student model obtains the clustering information of the target language with the help of  . For  analysis, we consider the consistency of recognition results and similarity score by teachers. The F1-score and similarity score of teachers are all higher in the higher  intervals, as shown in Figure 6c. The student model learns less from unreasonable results, and it can make more accurate entity recognition for the target language. In this paper, we propose an unsupervised multiple-task and multiple-teacher model for cross-lingual NER. The student model learns two source language patterns of entity recognition and entity similarity evaluation. Moreover, to guarantee the student learning performance, we also propose a weighting strategy to take into consideration the reliability of the teachers. Our experimental results show that the proposed model yields significant improvements on six target language datasets and outperforms the existing state-of-the-art approaches. This work is supported partly by the Fundamental Research Funds for the Central Universities and by the State Key Laboratory of Software Development Environment.
abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective abstain method abstain result objective objective result abstain other other other other other other other other other other other abstain other other other other abstain other other other other abstain objective abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain other abstain abstain abstain method abstain method abstain method abstain other abstain other abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain method abstain method abstain abstain method method method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective other