In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability. It has become a de facto trend to use a pretrained language model (Devlin et al., 2019; Dong et al., 2019; Yang et al., 2019b; Bao et al., 2020) for downstream NLP tasks. These models are typically pretrained with masked language modeling objectives, which learn to generate the masked tokens of an input sentence. In addition to monolingual representations, the masked language modeling task is effective for learning cross-lingual representations. By only using multilingual corpora, such pretrained models perform well on zero-shot cross-lingual transfer (Devlin et al., 2019; Conneau et al., 2020), i.e., fine-tuning with English training data while directly applying the model to other target languages. The cross-lingual transferability can be further improved by introducing external pre-training tasks using parallel corpus, such as translation language modeling (Conneau and Lample, 2019), and cross-lingual contrast (Chi et al., 2021b). However, previous cross-lingual pre-training based on masked language modeling usually requires massive computation resources, rendering such models quite expensive. As shown in Figure 1, our proposed  Equal contribution. XLM-E achieves a huge speedup compared with well-tuned pretrained models. In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two discriminative pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Rather than recovering masked tokens, the model learns to distinguish the replaced tokens in the corrupted input sequences. The two tasks build input sequences by replacing tokens in multilingual sentences, and translation pairs, respectively. We also describe the pretraining algorithm of our model, XLM-E, which is pretrained with the above two discriminative tasks. It provides a more compute-efficient and sample-efficient way for cross-lingual language model pretraining. We conduct extensive experiments on the XTREME cross-lingual understanding benchmark to evaluate and analyze XLM-E. Over seven datasets, our model achieves competitive results with the baseline models, while only using 1% of the computation cost comparing to XLM-R. In addition to the high computational efficiency, our model also shows the cross-lingual transferability that achieves a reasonably low transfer gap. We also show that the discriminative pre-training encourages universal representations, making the text representations better aligned across different languages. Our contributions are summarized as follows:  We explore ELECTRA-style tasks for cross-lingual language model pre-training, and pretrain XLM-E with both multilingual corpus and parallel data. We demonstrate that XLM-E greatly reduces the computation cost of cross-lingual pretraining. We show that discriminative pre-training tends to encourage better cross-lingual transferability. ELECTRAELECTRA (Clark et al., 2020b) introduces the replaced token detection task for language model pre-training, with the goal of distinguishing real input tokens from corrupted tokens. That means the text encoders are pretrained as discriminators rather than generators, which is different from the previous pretrained language models, such as BERT (De-vlin et al., 2019), that learn to predict the masked tokens. ELECTRA trains two Transformer (Vaswani et al., 2017) encoders, serving as generator and discriminator, respectively. The generator G is typically a small BERT model trained with the masked language modeling (MLM; Devlin et al. 2019) task. Consider an input sentence x = { x i } ni =1 containing n tokens. MLM first randomly selects a subset M  { 1 , . . . , n } as the positions to be masked, and construct the masked sentence x masked by replacing tokens in M with [MASK] . Then, the generator predicts the probability distributions of the masked tokens p G ( x | x masked ) . The loss function of the generator G is: LG ( x ;  G ) =  (cid:88) i M log p G ( x i | x masked ) . The discriminator D is trained with the replaced token detection task. Specifically, the discriminator takes the corrupted sentences x corrupt as input, which is constructed by replacing the tokens in M with the tokens sampled from the generator G : (cid:40) x corrupt i  p G ( x i | x masked ) , i  M x corrupt i = x i , i (cid:54) M (2) Then, the discriminator predicts whether x corrupt i is original or sampled from the generator. where z i represents the label of whether x corrupt i is the original token or the replaced one. The final loss function of ELECTRA is the combined loss of the generator and discriminator losses, LE = LG +  LD . Compared to generative pre-training, ELECTRA uses more model parameters and training FLOPs per step, because it contains a generator and a discriminator during pre-training. However, only the discriminator is used for fine-tuning on downstream tasks, so the size of the final checkpoint is similar to BERT-like models in practice. Figure 2 shows an overview of the two discriminative tasks used for pre-training XLM-E. Similar to ELECTRA described in Section 2, XLM-E has two Transformer components, i.e., generator and discriminator. The generator predicts the masked tokens given the masked sentence or translation pair, and the discriminator distinguishes whether the tokens are replaced by the generator. The pre-training tasks of XLM-E are multilingual replaced token detection (MRTD), and translation replaced token detection (TRTD). The multilingual replaced token detection task requires the model to distinguish real input tokens from corrupted multilingual sentences. Both the generator and the discriminator are shared across languages. The vocabulary is also shared for different languages. The task is the same as in monolingual ELECTRA pre-training (Section 2). The only difference is that the input texts can be in various languages. We use uniform masking to produce the corrupted positions. We also tried span masking (Joshi et al., 2019; Bao et al., 2020) in our preliminary experiments. The results indicate that span masking significantly weakens the generator's prediction accuracy, which in turn harms pre-training. Translation Replaced Token Detection Parallel corpora are easily accessible and proved to be effective for learning cross-lingual language models (Conneau and Lample, 2019; Chi et al., 2021b), while it is under-studied how to improve discriminative pre-training with parallel corpora. We introduce the translation replaced token detection task that aims to distinguish real input tokens from translation pairs. Given an input translation pair, the generator predicts the masked tokens in both languages. Consider an input translation pair ( e , f ) . We construct the input sequence by concatenating the translation pair as a single sentence. The loss function of the generator G is: LG ( e , f ;  G ) =  (cid:88) i M e log p G ( e i | [ e ; f ] masked )  (cid:88) i M f log p G ( f i | [ e ; f ] masked ) where [; ] is the operator of concatenation, and M e , M f stand for the randomly selected masked positions for e and f , respectively. This loss function is identical to the translation language modeling loss (TLM; Conneau and Lample 2019). The discriminator D learns to distinguish real input tokens from the corrupted translation pair. The corrupted translation pair ( e corrupt , f corrupt ) is constructed by replacing tokens with the tokens sampled from G with the concatenated translation pair as input. Formally, e corrupt is constructed by (cid:40) e corrupt i  p G ( e i | [ e ; f ] masked ) , i  M e e corrupt i = e i , i (cid:54) M e (4) The same operation is also used to construct f corrupt . Then, the loss function of the discriminator D can be written as LD ( e , f ;  D ) =  n e + n f (cid:88) i =1 log p D ( r i | [ e ; f ] corrupt ) (5) where r i represents the label of whether the i -th input token is the original one or the replaced one. The final loss function of the translation replaced token detection task is LG +  LD . The XLM-E model is jointly pretrained with the masked language modeling, translation language modeling, multilingual replaced token detection and the translation replaced token detection tasks. The overall training objective is to minimize L = LMLM ( x ;  G ) + LTLM ( e , f ;  G ) +  LMRTD ( x ;  D ) +  LTRTD ( e , f ;  D ) over large scale multilingual corpus X = { x } and parallel corpus P = { ( e , f ) } . the generator and the discriminator from scratch. Following Clark et al. (2020b), we make the generator smaller to improve the pre-training efficiency. We propose to use gated relative position bias in the self-attention mechanism. Given input tokens { x i } | x | i =1 , let { h i } | x | i =1 denote their hidden states in Transformer. The self-attention outputs {  h i } | x | i =1 are computed via: q i , k i , v i = h i WQ , h i WK , h i WV (6) a ij  exp { q i  k j  d k + r i  j } (7)  h i = | x | (cid:88) j =1 a ij v i (8) where r i  j represents gated relative position bias, each h i is linearly projected to a triple of query, key and value using parameter matrices WQ , WK , WV  R d h  d k , respectively. Inspired by the gating mechanism of Gated Recurrent Unit (GRU; Cho et al. 2014), we compute gated relative position bias r i  j via: g (update) , g (reset) =  ( q i  u ) ,  ( q i  v )  r i  j = wg (reset) d i  j r i  j = d i  j + g (update) d i  j + (1  g (update) ) r i  j where d i  j is learnable relative position bias, the vectors u , v  R d k are parameters,  is a sigmoid function, and w is a learnable value. Compared with relative position bias (Parikh et al., 2016; Raffel et al., 2020; Bao et al., 2020), the proposed gates take the content into consideration, which adaptively adjusts the relative position bias by conditioning on input tokens. Intuitively, the same distance between two tokens tends to play different roles in different languages. Data We use the CC-100 (Conneau et al., 2020) dataset for the replaced token detection task. CC-100 contains texts in 100 languages collected from the CommonCrawl dump. We use parallel corpora for the translation replaced token detection task, including translation pairs in 100 languages collected from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), WikiMatrix (Schwenk et al., 2019), and CCAligned (El-Kishky et al., 2020). Following XLM (Conneau and Lample, 2019), we sample multilingual sentences to balance the language distribution. Formally, consider the pretraining corpora in N languages with m j examples for the j -th language. The probability of using an example in the j -th language is p j = m j (cid:80) Nk =1 m k (9) The exponent  controls the distribution such that a lower  increases the probability of sampling examples from a low-resource language. In this paper, we set  = 0 . 7 . Model We use a Base-size 12 -layer Transformer (Vaswani et al., 2017) as the discriminator, with hidden size of 768 , and FFN hidden size of 3 , 072 . The generator is a 4 -layer Transformer using the same hidden size as the discriminator (Meng et al., 2021). See Appendix A for more details of model hyperparameters. Training We jointly pretrain the generator and the discriminator of XLM-E from scratch, using the Adam (Kingma and Ba, 2015) optimizer for 125K training steps. We use dynamic batching of approximately 1M tokens for each pre-training task. We set  , the weight for the discriminator objective to 50. The whole pre-training procedure takes about 1.7 days on 64 Nvidia A100 GPU cards. See Appendix B for more details of pre-training hyperparameters. We evaluate XLM-E on the XTREME (Hu et al., 2020b) benchmark, which is a multilingual multitask benchmark for evaluating cross-lingual understanding. The XTREME benchmark contains seven cross-lingual understanding tasks, namely part-of-speech tagging on the Universal Dependencies v2.5 (Zeman et al., 2019), NER named entity recognition on the Wikiann (Pan et al., 2017; Rahimi et al., 2019) dataset, cross-lingual natural language inference on XNLI (Conneau et al., 2018), cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019a), and cross-lingual question answering on MLQA (Lewis et al., 2020), XQuAD (Artetxe et al., 2020), and TyDiQA-GoldP (Clark et al., 2020a). Baselines We compare our XLM-E model with the cross-lingual language models pretrained with multilingual text, i.e., Multilingual BERT ( MBERT ; Devlin et al. 2019), M T5 (Xue et al., 2021), and XLM-R (Conneau et al., 2020), or pretrained with both multilingual text and parallel corpora, i.e., XLM (Conneau and Lample, 2019), INFOXLM (Chi et al., 2021b), and XLM-ALIGN (Chi et al., 2021c). The compared models are all in Base size. In what follows, models are considered as in Base size by default. Results We use the cross-lingual transfer setting for the evaluation on XTREME (Hu et al., 2020b), where the models are first fine-tuned with the English training data and then evaluated on the target languages. In Table 1, we report the accuracy, F1, or Exact-Match (EM) scores on the XTREME cross-lingual understanding tasks. The results are averaged over all target languages and five runs with different random seeds. We divide the pretrained models into two categories, i.e., the models pretrained on multilingual corpora, and the models pretrained on both multilingual corpora and parallel corpora. For the first setting, we pretrain XLM-E with only the multilingual replaced token detection task. From the results, it can be observed that XLM-E outperforms previous models on both settings, achieving the averaged scores of 67.6 and 69.3, respectively. Compared to XLM-R, XLM-E (w/o TRTD) produces an absolute 1.2 improvement on average over the seven tasks. For the second setting, compared to XLM-ALIGN , XLM-E produces an absolute 0.4 improvement on average. XLM-E performs better on the question answering Model XNLI MLQA XLM (reimplementation) 73.4 66.2 / 47.8  TLM 70.6 64.0 / 46.0 XLM-E 76.6 68.3 / 49.8  TRTD 75.1 67.8 / 49.7  TRTD  Gated relative position bias 75.2 67.4 / 49.2 Table 2: Ablation studies of XLM-E. Despite the effectiveness of XLM-E, our model requires substantially lower computation cost than XLM-R and XLM-ALIGN . A detailed efficiency analysis in presented in Section 4.5. For a deeper insight to XLM-E, we conduct ablation experiments where we first remove the TRTD task and then remove the gated relative position bias. Besides, we reimplement XLM that is pretrained with the same pre-training setup with XLM-E, i.e., using the same training steps, learning rate, etc. Table 2 shows the ablation results on XNLI and MLQA. Removing TRTD weakens the performance of XLM-E on both downstream tasks. On this basis, the results on MLQA further decline when removing the gated relative position bias. This demonstrates that XLM-E benefits from both TRTD and the gated relative position bias during pre-training. Besides, XLM-E substantially outperform XLM on both tasks. Notice that when removing the two components from XLM-E, our 6174 Model Size Params XNLI MLQA XLM-E Base 279M 76.6 68.3 / 49.8 XLM-E Large 840M 81.3 72.7 / 54.2 XLM-E XL 2.2B 83.7 76.2 / 57.9 XLM-R XL 3.5B 82.3 73.4 / 55.3 M T5 XL 3.7B 82.9 73.5 / 54.5 Table 3: Results of scaling-up the model size. model only requires a multilingual corpus, but still achieves better performance than XLM, which uses an additional parallel corpus. Scaling-up model size has shown to improve performance on cross-lingual downstream tasks (Xue et al., 2021; Goyal et al., 2021). We study the scal-ability of XLM-E by pre-training XLM-E models using larger model sizes. We consider two larger model sizes in our experiments, namely Large and XL. Detailed model hyperparameters can be found in Appendix A. As present in Table 3, XLM-EXL achieves the best performance while using significantly fewer parameters than its counterparts. Besides, scaling-up the XLM-E model size consistently improves the results, demonstrating the effectiveness of XLM-E for large-scale pre-training. We present a comparison of the pre-training resources, to explore whether XLM-E provides a more compute-efficient and sample-efficient way for pre-training cross-lingual language models. Table 4 compares the XTREME average score, the number of parameters, and the pre-training computation cost. Notice that INFOXLM and XLM-ALIGN are continue-trained from XLM-R, so the total training FLOPs are accumulated over XLM-R. Table 4 shows that XLM-E substantially reduces the computation cost for cross-lingual language model pre-training. Compared to XLM-R and XLM-ALIGN that use at least 9.6e21 training Model Tatoeba-14 Tatoeba-36 en  xx xx  en en  xx xx  en XLM-R 59.5 57.6 55.5 53.4 INFOXLM 80.6 77.8 68.6 67.3 XLM-E 74.4 72.3 65.0 62.3  TRTD 55.8 55.1 46.4 44.6 Table 5: Average accuracy@1 scores for Tatoeba cross-lingual sentence retrieval. FLOPs, XLM-E only uses 9.5e19 training FLOPs in total while even achieving better XTREME performance than the two baseline models. For the setting of pre-training with only multilingual corpora, XLM-E (w/o TRTD) also outperforms XLM-R using 6.3e19 FLOPs in total. This demonstrates the compute-effectiveness of XLM-E, i.e., XLM-E as a stronger cross-lingual language model requires substantially less computation resource. To explore whether discriminative pre-training improves the resulting cross-lingual representations, we evaluate our model on the sentence-level and word-level alignment tasks, i.e., cross-lingual sentence sentence retrieval and word alignment. We use the Tatoeba (Artetxe and Schwenk, 2019) dataset for the cross-lingual sentence retrieval task, the goal of which is to find translation pairs from the corpora in different languages. Tatoeba consists of English-centric parallel corpora covering 122 languages. Following Chi et al. (2021b) and Hu et al. (2020b), we consider two settings where we use 14 and 36 of the parallel corpora for evaluation, respectively. The sentence representations are obtained by average pooling over hidden vectors from a middle layer. Specifically, we use layer-7 for XLM-R and layer-9 for XLM-E. Then, the translation pairs are induced by the nearest neighbor search using the cosine similarity. Table 5 shows the average accuracy@1 scores under the two settings of Tatoeba for both the xx  en and en  xx directions. XLM-E achieves 74.4 and 72.3 accuracy scores for Tatoeba-14, and 65.0 and 62.3 accuracy scores for Tatoeba-36, providing notable improvement over XLM-R. XLM-E performs slightly worse than INFOXLM. We believe the cross-lingual contrast (Chi et al., 2021b) task explicitly learns the sentence representations, which makes INFOXLM more effective for the cross-lingual sentence retrieval task. For the word-level alignment, we use the word alignment datasets from EuroParl 1 , WPT2003 2 , and WPT2005 3 , containing 1,244 translation pairs annotated with golden alignments. The predicted alignments are evaluated by alignment error rate (AER; Och and Ney 2003): AER = 1  | A  S | + | A  P | | A | + | S | (10) where A, S, and P stand for the predicted alignments, the annotated sure alignments, and the annotated possible alignments, respectively. In Table 6 we compare XLM-E with baseline models, i.e., fast align (Dyer et al., 2013), XLM-R, and XLM-ALIGN . The resulting word alignments are obtained by the optimal transport method (Chi et al., 2021c), where the sentence representations are from the 9 -th layer of XLM-E. Over the four language pairs, XLM-E achieves lower AER scores than the baseline models, reducing the average AER from 21 . 05 to 19.32. It is worth mentioning that our model requires substantial lower computation costs than the other cross-lingual pretrained language models to achieve such low AER scores. See the detailed training efficiency analysis in Section 4.5. It is worth mentioning that XLM-E shows notable improvements over XLM-E (w/o TRTD) on both tasks, demonstrating that the translation replaced token detection task is effective for cross-lingual alignment. We evaluate the word-level and sentence-level representations over different layers to explore As shown in Figure 3, we illustrate the accu-racy@1 scores of XLM-E and XLM-R on Tatoeba cross-lingual sentence retrieval, using sentence representations from different layers. For each layer, the final accuracy score is averaged over all the 36 language pairs in both the xx  en and en  xx directions. From the figure, it can be observed that XLM-E achieves notably higher averaged accuracy scores than XLM-R for the top layers. The results of XLM-E also show a parabolic trend across layers, i.e., the accuracy continuously increases before a specific layer and then continuously drops. This trend is also found in other cross-lingual language models such as XLM-R and XLM-Align (Jalili Sabet et al., 2020; Chi et al., 2021c). Different from XLM-R that achieves the highest accuracy of 54.42 at layer-7, XLM-E pushes it to layer-9, achieving an accuracy of 63.66. At layer-10, XLM-R only obtains an accuracy of 43.34 while XLM-E holds the accuracy score as high as 57.14. (AER) scores of XLM-E and XLM-R on the word alignment task. We use the hidden vectors from different layers to perform word alignment, where layer-0 stands for the embedding layer. The final AER scores are averaged over the four test sets in different languages. Figure 4 shows a similar trend to that in Figure 3, where XLM-E not only provides substantial performance improvements over XLM-R, but also pushes the best-performance layer to a higher layer, i.e., the model obtains the best performance at layer-9 rather than a lower layer such as layer-7. On both tasks, XLM-E shows good performance for the top layers, even though both XLM-E and XLM-R use the Transformer (Vaswani et al., 2017) architecture. Compared to the masked language modeling task that encourages the top layers to be language-specific, discriminative pre-training makes XLM-E producing better-aligned text representations at the top layers. It indicates that the cross-lingual discriminative pre-training encourages universal representations inside the model. We analyze the cross-lingual transfer gap (Hu et al., 2020b) of the pretrained cross-lingual language models. The transfer gap score is the difference between performance on the English test set and the average performance on the test set in other languages. This score suggests how much end task knowledge has not been transferred to other languages after fine-tuning. A lower gap score indicates better cross-lingual transferability. Table 7 compares the cross-lingual transfer gap scores on five of the XTREME tasks. We notice that XLM-E obtains the lowest gap score only on PAWS-X. Nonetheless, it still achieves reasonably low gap scores on the other tasks with such low computation cost, demonstrating the cross-lingual transferability of XLM-E. We believe that it is more difficult to achieve the same low gap scores when the model obtains better performance. Learning self-supervised tasks on large-scale multilingual texts has proven to be effective for pretraining cross-lingual language models. Masked language modeling (MLM; Devlin et al. 2019) is typically used to learn cross-lingual encoders such as multilingual BERT (mBERT; Devlin et al. 2019) and XLM-R (Conneau et al., 2020). The cross-lingual language models can be further improved by introducing external pre-training tasks using parallel corpora. XLM (Conneau and Lample, 2019) introduces the translation language modeling (TLM) task that predicts masked tokens from concatenated translation pairs. ALM (Yang et al., 2020) utilizes translation pairs to construct code-switched sequences as input. InfoXLM (Chi et al., 2021b) considers an input translation pair as cross-lingual views of the same meaning, and proposes a cross-lingual contrastive learning task. Several pre-training tasks utilize the token-level alignments in parallel data to improve cross-lingual language models (Cao et al., 2020; Zhao et al., 2021; Hu et al., 2020a; Chi et al., 2021c). In addition, parallel data are also employed for cross-lingual sequence-to-sequence pre-training. XNLG (Chi et al., 2020) presents cross-lingual masked language modeling and cross-lingual auto-encoding for cross-lingual natural language generation, and achieves the cross-lingual transfer for NLG tasks. VECO (Luo et al., 2020) utilizes cross-attention MLM to pretrain a variable cross-lingual language model for both NLU and NLG. mT6 (Chi et al., 2021a) improves mT5 (Xue et al., 2021) by learning the translation span corruption task on parallel data. LM (Ma et al., 2021) proposes to align pretrained multilingual encoders to improve cross-lingual sequence-to-sequence pre-training. We introduce XLM-E, a cross-lingual language model pretrained by ELECTRA-style tasks. Specifically, we present two pre-training tasks, i.e., multilingual replaced token detection, and translation replaced token detection. XLM-E outperforms baseline models on cross-lingual understanding tasks although using much less computation cost. In addition to improved performance and computational efficiency, we also show that XLM-E 6177 obtains the cross-lingual transferability with a reasonably low transfer gap. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems , pages 70577067. Curran Associates, Inc. 6178 Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Our work introduces ELECTRA-style tasks for cross-lingual language model pre-training, which requires much less computation cost than previous models and substantially reduces the energy cost. Heyan Huang is the corresponding author. Zewen Chi, Xian-Ling Mao, and Heyan Huang are supported by National Key R&D Plan (No. 2018YFB1005100), National Natural Science Foundation of China (No. U19B2020, 62172039, 61732005, 61602197 and L1924068), the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005), and in part by CCF-AFSG Research Fund under Grant No.RF20210005, and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL).
abstain method method result abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain abstain abstain abstain method result result result objective objective result other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain other method abstain abstain method abstain method method abstain abstain method abstain method abstain abstain method method method abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result other other other other other other other other other other other other abstain method abstain result abstain abstain abstain abstain abstain method other other