While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks (Con-neau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/ google/LaBSE . In this paper, we systematically explore using pretraining language models in combination with the best of existing methods for learning cross-lingual sentence embeddings. Such embeddings are useful for clustering, retrieval, and modular use of text representations for downstream tasks. While  Equal contributions. existing cross-lingual sentence embedding models incorporate large transformer models, using large pretrained language models is not well explored. Rather in prior work, encoders are trained directly on translation pairs (Artetxe and Schwenk, 2019b; Guo et al., 2018; Yang et al., 2019a), or on translation pairs combined with monolingual input-response prediction (Chidambaram et al., 2019; Yang et al., 2019b). In our exploration, as illustrated in figure 1, we make use of dual-encoder models, which have been demonstrated as an effective approach for learning bilingual sentence embeddings (Guo et al., 2018; Yang et al., 2019a). However, diverging from prior work, rather than training encoders from scratch, we investigate using pre-trained encoders based on large language models. We contrast models with and without additive margin softmax (Yang et al., 2019a) 1 . Figure 2 illustrates where our work stands (shaded) in the field of LM pre-training and sentence embedding learning. Our massively multilingual models outperform the previous state-of-the-art on large bi-text retrieval tasks including the United Nations (UN) 1 We also investigate the impact of mining hard negatives (Guo et al., 2018), but found it doesn't provide additional gain on top of other approaches. See supplemental material for details. corpus (Ziemski et al., 2016) and BUCC (Zweigen-baum et al., 2018). Table 1 compares our best model with other recent multilingual work. Both the UN corpus and BUCC cover resource rich languages (fr, de, es, ru, and zh). We further evaluate our models on the Tatoeba retrieval task (Artetxe and Schwenk, 2019b) that covers 112 languages. Compare to LASER (Artetxe and Schwenk, 2019b), our models perform significantly better on low-resource languages, boosting the overall accuracy on 112 languages to 83.7%, from the 65.5% achieved by the previous state-of-art. Surprisingly, we observe our models performs well on 30+ Tatoeba languages for which we have no explicit monolingual or bilingual training data. Finally, our embeddings perform competitively on the SentEval sentence embedding transfer learning benchmark (Conneau and Kiela, 2018). The contributions of this paper are:  A novel combination of pre-training and dual-encoder finetuning to boost translation ranking performance, achieving a new state-of-the-art on bi-text mining. A publicly released multilingual sentence embedding model spanning 109+ languages . Thorough experiments and ablation studies to understand the impact of pre-training, negative sampling strategies, vocabulary choice, data quality, and data quantity. Dual encoder models are an effective approach for learning cross-lingual embeddings (Guo et al., 2018; Yang et al., 2019a). Such models consist of paired encoding models that feed a scoring function. The source and target sentences are encoded separately. Sentence embeddings are extracted from each encoder. Cross-lingual embeddings are trained using a translation ranking task with in-batch negative sampling: L =  1 NN (cid:88) i =1 log e  ( x i ,y i ) e  ( x i ,y i ) + (cid:80) Nn =1 ,n (cid:54) = i e  ( x i ,y n ) (1) The embedding space similarity of x and y is given by  ( x, y ) , typically  ( x, y ) = xy T . The loss attempts to rank y i , the true translation of x i , over all N  1 alternatives in the same batch. Notice that L is asymmetric and depends on whether the softmax is over the source or the target sentences. For bidirectional symmetry, the final loss can sum the source-to-target, L , and target-to-source, L (cid:48) , losses (Yang et al., 2019a):  L = L + L (cid:48) (2) Dual encoder models trained using a translation ranking loss directly maximize the similarity of translation pairs in a shared embedding space. Additive margin softmax extends the scoring function  by introducing margin m around positive pairs (Yang et al., 2019a): The margin, m , improves the separation between translations and nearby non-translations. Using  (cid:48) ( x i , y j ) with the bidirectional loss  L s , we obtain the additive margin loss L =  1 NN (cid:88) i =1 e  ( x i ,y i )  m e  ( x i ,y i )  m + (cid:80) Nn =1 ,n (cid:54) = i e  ( x i ,y n ) (4) 879 2.2 MLM and TLM Pre-training Only limited prior work has combined dual encoders trained with a translation ranking loss with encoders initialized using large pre-trained language models (Yang et al., 2021). We contrast using a randomly initialized transformer, as was done in prior work (Guo et al., 2018; Yang et al., 2019a), with using a large pre-trained language model. For pre-training, we combined Masked language modeling (MLM) (Devlin et al., 2019) and Translation language modeling (TLM) (Con-neau and Lample, 2019). MLM is a variant of a cloze task, whereby a model uses context words surrounding a [MASK] token to try to predict what the [MASK] word should be. TLM extends this to the multilingual setting by modifying MLM training to include concatenated translation pairs. Multilingual pre-trained models such as mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019) and XLM-R (Conneau et al., 2019) have led to exceptional gains across a variety of cross-lingual natural language processing tasks (Hu et al., 2020). However, without a sentence level objective, they do not directly produce good sentence embeddings. As shown in Hu et al. (2020), the performance of such models on bitext retrieval tasks is very weak, e.g XLM-R Large gets 57.3% accuracy on a selected 37 languages 2 from the Tatoeba dataset compared to 84.4% using LASER (see performance of more models in table 5). We contribute a detailed exploration that uses pre-trained language models to produce useful multilingual sentence embeddings. Monolingual Data We collect monolingual data from CommonCrawl 4 and Wikipedia 5 . We use the 2019-35 version of CommonCrawl with heuristics from Raffel et al. (2019) to remove noisy text. Additionally, we remove short lines < 10 characters and those > 5000 characters. 6 The wiki 2 The number is counted from official evaluation script despite the original paper says 33 languages. data is extracted from the 05-21-2020 dump using WikiExtractor 7 . An in-house tool splits the text into sentences. The sentences are filtered using a sentence quality classifier. 8 After filtering, we obtain 17B monolingual sentences, about 50% of the unfiltered version. The monolingual data is only used in custom pre-training. Bilingual Translation Pairs The translation corpus is constructed from web pages using a bitext mining system similar to the approach described in Uszkoreit et al. (2010). The extracted sentence pairs are filtered by a pre-trained contrastive-data-selection (CDS) scoring model (Wang et al., 2018). Human annotators manually evaluate sentence pairs from a small subset of the harvested pairs and mark the pairs as either GOOD or BAD translations. The data-selection scoring model threshold is chosen such that 80% of the retained pairs from the manual evaluation are rated as GOOD. We limit the maximum number of sentence pairs to 100 million for each language to balance the data distribution. Many languages still have far fewer than 100M sentences. The final corpus contains 6B translation pairs. 9 The translation corpus is used for both dual encoder training and custom pre-training. In this section, we describe the training details for the dual encoder model. A transformer encoder is used in all experiments (Vaswani et al., 2017). We train two versions of the model, one uses the public BERT multilingual cased vocab with vocab size 119,547 and a second incorporates a customized vocab extracted over our training data. For the customized vocab, we employ a wordpiece tok-enizer (Sennrich et al., 2016), with a cased vocabulary extracted from the training set using TF Text. 10 The language smoothing exponent for the vocab generation tool is set to 0.3 to counter imbalances in the amount of data available per language. The final vocabulary size is 501,153. The encoder architecture follows the BERT Base model, with 12 transformer blocks, 12 attention 7 https://github.com/attardi/ wikiextractor 8 The quality classifier is trained using sentences from the main content of webpages as positives and text from other areas as negatives. heads and 768 per-position hidden units. The encoder parameters are shared for all languages. Sentence embeddings are extracted as the l 2 normalized [CLS] token representations from the last transformer block. 11 Our models are trained on Cloud TPU V3 with 32-cores using a global batch size of 4096 with a maximum sequence length of 128, using the AdamW (Loshchilov and Hutter, 2019) optimizer with initial learning rate 1e-3, and linear weight decay. We train for 50k steps for pre-trained models, and 500k steps for models without pre-training. We observe that additional training did not change the performance significantly. The default margin value for additive margin softmax is set to 0.3. Hy-perparameters are tuned on a held-out development set. Cross-lingual embedding models trained with in-batch negative samples benefit from large training batch sizes (Guo et al., 2018). Resource intensive models like BERT, are limited to small batch sizes due to memory constraints. While data-parallelism does allow us to increase the global batch size by using multiple accelerators, the batch-size on individual cores remains small. For example, a 4096 batch run across 32 cores results in a local batch size of 128, with each example then only receiving 127 negatives. sam-11 During training, the sentence embeddings after normalization are multiplied by a scaling factor. Following Chidambaram et al. (2018), we set the scaling factor to 10. We observe that the scaling factor is important for training a dual encoder model with the normalized embeddings. pling , which is illustrated in figure 3. 12 Under this strategy each core encodes its assigned sentences and then the encoded sentence representations from all cores are broadcast as negatives to the other cores. This allows us to fully realize the benefits of larger batch sizes while still distributing the computationally intensive encoding work across multiple cores. Note the dot-product scoring function makes it efficient to compute the pairwise scores in the same batch with matrix multiplication. In figure 3, the value in the grids indicates the ground truth labels, with all positive labels located in diagonal grids. A softmax function is applied on each row. The encoder is pre-trained with Masked Language Model (MLM) (Devlin et al., 2019) and Translation Language Model (TLM) (Conneau and Lample, 2019) 13 training on the monolingual data and bilingual translation pairs, respectively. For an L layer transformer encoder, we train using a 3 stage progressive stacking algorithm (Gong et al., 2019), where we first learn a L 4 layers model and then L 2 layers and finally all L layers. The parameters of the models learned in the earlier stages are copied to the models for the subsequent stages. Pre-training uses TPUv3 with 512-cores and a batch size of 8192. The max sequence length is set to 512 and 20% of tokens (or 80 tokens at most) per sequence are masked for MLM and TLM predictions. For the three stages of progressive stacking, 12 While our experiments use TPU accelerators, the same strategy can also be applied to models trained on GPU. We evaluate models on three bitext retrieval tasks: United Nations (UN), Tatoeba, and BUCC. All tasks are to retrieve the correct English translation for each non-English sentence. United Nations (UN) contains 86,000 sentence aligned bilingual documents over five language pairs: en-fr, en-es, en-ru, en-ar and en-zh (Ziemski et al., 2016). A total of 11.3 million 14 aligned sentence pairs can be extract from the document pairs. The large pool of translation candidates makes this data set particularly challenging. Tatoeba evaluates translation retrieval over 112 languages (Artetxe and Schwenk, 2019b). The dataset contains up to 1,000 sentences per language along with their English translations. We evaluate performance on the original version covering all 112 languages, and also the 36 languages version from the XTREME benchmark (Hu et al., 2020). BUCC is a parallel sentence mining shared task (Zweigenbaum et al., 2018). We use the 2018 shared task data, containing four language pairs: fren, de-en, ru-en and zh-en. For each pair, the task provides monolingual corpora and gold true translation pairs. The task is to extract translation pairs from the monolingual data, which are evaluated against the ground truth using F1. Since the ground truth for the BUCC test data is not released, we follow prior work using the BUCC training set for evaluation rather than training (Yang et al., 2019b; Hu et al., 2020). Sentence embedding cosine similarity is used to identify the translation pairs. 15 4.2 Downstream Classification We also evaluate the transfer performance of multilingual sentence embeddings on downstream classification tasks from the SentEval benchmark (Con-neau and Kiela, 2018). We evaluate on select tasks from SentEval including: ( MR ) movie reviews (Pang and Lee, 2005)), ( SST ) sentiment 14 About 9.5 million after de-duping. 15 Reranking models can further improve performance (e.g. margin based scorers (Artetxe and Schwenk, 2019a) and BERT based classifiers (Yang et al., 2019a)). However, this is tangential to assessing the raw embedding retrieval performance. analysis (Socher et al., 2013), ( TREC ) question-type (Voorhees and Tice, 2000), ( CR ) product reviews (Hu and Liu, 2004), ( SUBJ ) subjectiv-ity/objectivity (Pang and Lee, 2004), ( MPQA ) opinion polarity (Wiebe et al., 2005), and ( MRPC ) paraphrasing detection (Dolan et al., 2004). While SentEval is English only, we make use of this benchmark in order to directly compare to prior work on sentence embedding models. Table 2 shows the performance on the UN and Tatoeba bitext retrieval tasks and compares against the prior state-of-the-art bilingual models Yang et al. (2019a), LASER (Artetxe and Schwenk, 2019b), and the multilingual universal sentence encoder ( m -USE) (Yang et al., 2019b) 16 . Row 1-3 show the performance of baseline models, as reported in the original papers. Row 4-7 shows the performance of models that use the public mBERT vocabulary. The baseline model shows reasonable performance on UN ranging from 57%-71% P@1. It also perform well on Tatoeba with 92.8% and 79.1% accuracy for the 36 language group and all languages, respectively. Adding pre-training both helps models converge faster (see details in section 6.2) and improves performance on the UN retrieval task using both vocabularies. Pre-training also helps on Taoeba, but only using the customized vocabulary. 17 Additive margin softmax significantly improves the performance on all model variations. The last two rows contain models using the customized vocab. Both of them are trained with additive margin softmax given the strong evidence from the experiments above. Both models outperform the mBERT vocabulary based models, and the pre-trained model performs best of all. The top model (Base w/ Customized Vocab + AMS + PT) achieves a new state-of-the-art on 3 of the 4 languages, with P@1 91.1, 88.3, 90.8 for en-es, en-fr, en-ru, respectively. It reaches 87.7 on zh-en, only 0.2 lower than the best bilingual en-zh model and nearly 9 points better than the previous best multilingual model . On Tatoeba, the best model also outperform the baseline model by a large margin, with +10.6 accuracy on the 36 language group 16 universal-sentence-encoder-multilingual-large/3 17 The coverage of the public mBERT vocabulary on the tail languages is bad with many [UNK] tokens for such languages, e.g. the [UNK] token rate is 71% for language si , which could be the reason pre-training doesn't help on the tatoeba task. from XTREME and +18.2 on all languages. It is worth noting that all our models perform similarly on Tatoeba but not on UN. This suggests it is necessary to evaluate on large scale bitext retrieval tasks to better discern differences between competing models. For the rest of the paper we refer to LaBSE as the best performing model here, Base w/ Customized Vocab + AMS + PT , unless otherwise specified. Table 3 provides LaBSE's retrieval performance on BUCC, comparing against strong baselines from Artetxe and Schwenk (2019a) and Yang et al. (2019a). Following prior work, we perform both forward and backward retrieval. Forward retrieval treats en as the target and the other language as the source, and backward retrieval is vice versa. LaBSE not only systematically outperforms prior work but also covers all languages within a single model. The previous state-of-the-art required four separate bilingual models (Yang et al., 2019a). Table 4 gives the transfer performance achieved by LaBSE on the SentEval benchmark (Conneau and Kiela, 2018), comparing against other state-of-the-art sentence embedding models. Despite its massive language coverage in a single model, LaBSE still obtains competitive transfer performance with monolingual English sentence embedding models and the 16 language m -USE model. The above experiments show that additive margin softmax is a critical factor in learning good crosslingual embeddings, which is aligned with the find-ings from Yang et al. (2019a). We further investi-0.1 0.0 0.1 0.2 0.3 0.4 0.5 Margin value 50 60 70 80 90 100 P @ 1 ( % ) UN P@1 (Averaged) with different margin value Base w/ mBERT vocab + AMS Base w/ mBERT vovab + AMS + PT Base w/ Customized vocab + AMS + PT (LaBSE) Figure 4: Average P@1 (%) on UN retrieval task of models trained with different margin values. gate the effect of margin size on our three model variations, as shown in figure 4. The model with an additive margin value 0 performs poorly on the UN task with  60 average P@1 across all three model variations. With a small margin value of 0.1, the model improves significantly compare to no margin with mid 70 to mid 80 average P@1. Consistently across models, increasing the margin value improves performance until it reaches 0.3. To better understand the effect of MLM/TLM pretraining on the final LaBSE model, we explore training a variant of this model using our customized vocab but without pre-training. The results are shown in figure 5. We experiment with varying the number of training steps for both models, including: 50k, 100K, 200K, and 500K steps. A model with pre-trained encoders achieves excellent performance when trained for only 50K steps and further training doesn't increase the performance significantly. However, the model without pre-training performs poorly when only trained 50k steps. Its performance increases with additional steps and approaches the model with pre-training at 500k steps. The overall performance is, how-883 Models fr-en de-en ru-en zh-en P R F P R F P R F P R FF o r w a r d Artetxe and Schwenk (2019a) 82.1 74.2 78.0 78.9 75.1 77.0 ---Yang et al. (2019a) 86.7 85.6 86.1 90.3 88.0 89.2 84.6 91.1 87.7 86.7 90.9 88.8 LaBSE 86.6 90.9 88.7 92.3 92.7 92.5 86.1 91.9 88.9 88.2 89.7 88.9 B ac k w a r d Artetxe and Schwenk (2019a) 77.2 72.7 74.7 79.0 73.1 75.9 ---Yang et al. (2019a) 83.8 85.5 84.6 89.3 87.7 88.5 83.6 90.5 86.9 88.7 87.5 88.1 LaBSE 87.1 88.4 87.8 91.3 92.7 92.0 86.3 90.7 88.4 87.8 90.3 89.0 Table 3: [P]recision, [R]ecall and [F]-score of BUCC training set score with cosine similarity scores. ever, still slightly worse. Moreover, further training past 500k steps doesn't increase the performance significantly. Pre-training thus both improves performance and dramatically reduces the amount of parallel data required. Critically, the model sees 1B examples at 500K steps, while the 50K model only sees 200M examples. 18 6.3 Low Resource Languages and Languages without Explicit Training Data We evaluate performance through further experiments on Tatoeba for comparison to prior work and 18 We note that it is relative easy to get 200M parallel examples for many languages from public sources like Paracrawl, TED58, while obtaining 1B examples is generally much more challenging. to identify broader trends. Besides the 36 language group and all-languages group, two more groups of 14 languages (selected from the languages covered by m -USE), and 82 languages (covered by the LASER training data) are evaluated. Table 5 provides the macro-average accuracy achieved by LaBSE on the four language groupings drawn from Tatoeba, comparing against LASER and m -USE. All three models perform well on the 14 major languages support by m -USE, with each model achieving an average accuracy > 93%. Both LaBSE and LASER perform moderately better than m -USE, with an accuracy of 95.3%. As more languages are included, the averaged accuracy for both LaBSE and LASER decreases, but with a notably more rapid decline for LASER. LaBSE systematically outperforms LASER on the groups of 36 languages (+10.6%), 82 languages (+11.4%), and 112 languages (+18.2%). Figure 6 provides the Tatoeba accuracies for languages where we don't have any explicit training data. There are a total of 30+ such languages. The performance is surprisingly good for most of the languages with an average accuracy around 60%. Nearly one third of them have accuracy greater than 75%, and only 7 of them have accuracy lower than 25%. One possible reason is that language mapping is done manually and some languages are close to those languages with training data but are treated differently according to ISO-639 standards. Additionally, since automatic language detection is used, some limited amount of data for the missing languages might be included during training. We suspect that the well performing zero-shot languages are close to some language(s) that we have in the training data. For example, yue and wuu are related to zh (Chinese) and fo has similarities to is (ICELANDIC). Multilingual generalization across so many languages is only possible due to 884 Model 14 Langs 36 Langs 82 Langs All Langs m -USE Trans. The Semantic Textual Similarity (STS) benchmark (Cer et al., 2017) measures the ability of models to replicate fine-grained human judgements of pairwise English sentence similarity. Models are scored according to their Pearson correlation, r , on gold labels ranging from 0, unrelated meaning, to 5, semantically equivalent, with intermediate values capturing carefully defined degrees of meaning overlap. STS is used to evaluate the quality of sentence-level embeddings by assessing the degree to which similarity between pairs of sentence embeddings aligns with human perception of sentence meaning similarity. Table 6 reports performance on the STS benchmark for LaBSE versus existing sentence embedding models. Following prior work, the semantic similarity of a sentence pair according to LaBSE is computed as the arccosine distance between the pair's sentence embeddings. 19 For comparison, we include numbers for SentenceBERT when it is fine-tuned on the STS task as well as ConvEmbed when an additional affine transform is trained to fit the embeddings to STS. We observe that LaBSE performs worse on pairwise English semantic similarity than other sentence embedding models. We sus-19 Within prior work, m -USE, USE and ConvEmbed use arccos distance to measure embedding space semantic similarity, while InferSent and SentenceBERT use cosine similarity. pect training LaBSE on translation pairs biases the model to excel at detecting meaning equivalence, but not at distinguishing between fine grained degrees of meaning overlap. Recently, Reimers and Gurevych (2020) demonstrated that an English sentence embedding model can be distilled to a multilingual student model using a language alignment loss. The distilled model performs well on multilingual STS benchmarks, but underperforms on bitext retrieval tasks when compared to state-of-the-art models. Our approach is complimentary and can be combined with their method to distill better student models. We use the LaBSE model to mine parallel text from CommonCrawl, a large-scale multilingual web corpus, and then train NMT models on the mined data. We experiment with two language pairs: English-to-Chinese (en-zh) and English-to-German (en-de). We mine translations from monolingual CommonCrawl data processed as described above for self-supervised MLM pretraining. After processing, there are 1.17B, 0.6B, 7.73B sentences for Chinese (zh), German (de), and English (en), respectively. LaBSE embeddings are used to pair each non-English sentence with its nearest English neighbor, dropping pairs with a similarity score < 0 . 6 . 20 For en-de and en-zh, we train a model with Transformer-Big (Vaswani et al., 2017) in the following way: First we train the model on the mined data as is for 120k steps with batch size 10k. Then we select the best 20% using Wang 20 The threshold 0.6 is selected by manually inspecting sampled data. We found pairs greater or equal to this threshold are likely to be translations or partial translations of each other. This results in 715M and 302M sentence pairs for en-zh and en-de, respectively. Note that the pairs may still be noisy, which is why we perform additional filtering before training NMT models (Wang et al., 2018) . et al. (2018)'s data selection method, and train for another 80k steps. Results in table 7 show the effectiveness of the mined training data. By referencing previous results (Edunov et al., 2018), we see that the model using the en-de mined data yields performance that is only 2.8 BLEU away from performance of the best system that made use of the official WMT17 en-de parallel data. Compare to prior en-zh results (Sennrich et al., 2017), we see that our model using mined en-zh training data is as good as a WMT17 NMT model that is trained on the official WMT en-zh parallel data. The table also gives BLEU performance on the TED test set (Qi et al., 2018), with performance of models trained on our mined training data being comparable with models trained using CCMatrix (Schwenk et al., 2019). 21 8 Conclusion This paper presents a language-agnostic BERT sentence embedding (LaBSE) model supporting 109 languages. The model achieves state-of-the-art performance on various bi-text retrieval/mining tasks compare to the previous state-of-the-art, while also providing increased language coverage. We show the model performs strongly even on those languages where LaBSE doesn't have any explicit training data, likely due to language similarity and the massively multilingual natural of the model. Extensive experiments show additive margin softmax is a key factor for training the model, parallel data quantity matters, but the effect of increased amounts of parallel data diminishes when a pre-trained language model is used. The pretrained model is released at https://tfhub. dev/google/LaBSE . 21 CCMatrix is another dataset contains billions of parallel sentences mined from CommonCrawl using a embedding based mining approach, with an additional cleaning step. We thank our teammates from Descartes, Translate and other Google groups for their feedback and suggestions. Special thanks goes to Sidharth Mud-gal, and Jax Law for help with data processing; as well as Jialu Liu, Tianqi Liu, Chen Chen, and Anosh Raj for help on BERT pretraining.
abstain abstain result abstain result other objective abstain abstain abstain abstain abstain objective abstain method abstain abstain abstain method abstain method result result method objective objective objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method method abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain other abstain abstain other other