Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space. However, previous works mostly adopt in-batch negatives or sample from training data at random. Such a way may cause the sampling bias that improper negatives ( e.g., false negatives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space. To address it, we present a new framework DCLR (Debiased Contrastive Learning of unsupervised sentence Representations) to alleviate the influence of these improper negatives. In DCLR, we design an instance weighting method to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. Experiments on seven semantic textual similarity tasks show that our approach is more effective than competitive baselines. Our code and data are publicly available at the link: https: //github.com/RUCAIBox/DCLR . As a fundamental task in the natural language processing (NLP) field, unsupervised sentence representation learning (Kiros et al., 2015; Hill et al., 2016) aims to derive high-quality sentence representations that can benefit various downstream tasks, especially for low-resourced domains or computationally expensive tasks, e.g., zero-shot text semantic matching (Qiao et al., 2016), large-scale semantic similarity comparison (Agirre et al., 2015), and document retrieval (Le and Mikolov, 2014). mantic representation approach, achieving remarkable performance on various NLP tasks. However, several studies have found that the native sentence representations derived by PLMs are not uniformly distributed with respect to directions, but instead occupy a narrow cone in the vector space (Ethayarajh, 2019), which largely limits their expressiveness. To address this issue, contrastive learning (Chen et al., 2020) has been adopted to refine PLM-derived sentence representations. It pulls semantically-close neighbors together to improve the alignment, while pushes apart non-neighbors for the uniformity of the whole representation space. In the learning process, both positive and negative examples are involved in contrast with the original sentence. For positive examples, previous works apply data augmentation strategies (Yan et al., 2021) on the original sentence to generate highly similar variations. While, negative examples are commonly sampled from the batch or training data ( e.g., in-batch negatives (Gao et al., 2021)) at random, due to the lack of ground-truth annotations for negatives. Although such a negative sampling way is simple and convenient, it may cause sampling bias and affect the sentence representation learning. First, the sampled negatives are likely to be false negatives that are indeed semantically close to the 6120 original sentence. As shown in Figure 1, given an input sentence, about half of in-batch negatives have a cosine similarity above 0.7 with the original sentence based on the SimCSE model (Gao et al., 2021). It is likely to hurt the semantics of the sentence representations by simply pushing apart these sampled negatives. Second, due to the anisotropy problem (Ethayarajh, 2019), the representations of sampled negatives are from the narrow representation cone spanned by PLMs, which cannot fully reflect the overall semantics of the representation space. Hence, it is sub-optimal to only rely on these representations for learning the uniformity objective of sentence representations. To address the above issues, we aim to develop a better contrastive learning approach with debiased negative sampling strategies.The core idea is to improve the random negative sampling strategy for alleviating the sampling bias problem. First, in our framework, we design an instance weighting method to punish the sampled false negatives during training. We incorporate a complementary model to evaluate the similarity between each negative and the original sentence, then assign lower weights for negatives with higher similarity scores. In this way, we can detect semantically-close false negatives and further reduce their influence. Second, we randomly initialize new negatives based on random Gaussian noises to simulate sampling within the whole semantic space, and devise a gradient-based algorithm to optimize the noise-based negatives towards the most nonuniform points. By learning to contrast with the nonuniform noise-based negatives, we can extend the occupied space of sentence representations and improve the uniformity of the representation space. To this end, we propose DCLR , a general framework towards Debiased Contrastive Learning of unsupervised sentence Representations. In our approach, we first initialize the noise-based negatives from a Gaussian distribution, and leverage a gradient-based algorithm to update the new negatives by considering the uniformity of the representation space. Then, we adopt the complementary model to produce the weights for these noise-based negatives and randomly sampled negatives, where the false negatives will be punished. Finally, we augment the positive examples via dropout (Gao et al., 2021) and combine them with the above weighted negatives for contrastive learning. We demonstrate that our DCLR outperforms a number of competitive baselines on seven semantic textual similarity (STS) tasks using BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). (1) To our knowledge, our approach is the first attempt to reduce the sampling bias in contrastive learning of unsupervised sentence representations. (2) We propose DCLR, a debiased contrastive learning framework that incorporates an instance weighting method to punish false negatives and generates noise-based negatives to guarantee the uniformity of the representation space. (3) Experimental results on seven semantic textual similarity tasks show the effectiveness of our framework. Sentence Representation Learning. Learning universal sentence representations (Kiros et al., 2015; Hill et al., 2016) is the key to the success of various downstream tasks. Previous works can be roughly categorized into supervised (Conneau et al., 2017; Cer et al., 2018) and unsupervised approaches (Hill et al., 2016; Li et al., 2020). Supervised approaches rely on annotated datasets ( e.g., NLI (Bowman et al., 2015; Williams et al., 2018)) to train the sentence encoder (Cer et al., 2018; Reimers and Gurevych, 2019). Unsupervised approaches consider deriving sentence representations without labeled datasets, e.g., pooling word2vec embeddings (Mikolov et al., 2013). Recently, to leverage the strong potential of PLMs (Devlin et al., 2019), several works propose to alleviate the anisotropy problem (Ethayarajh, 2019; Li et al., 2020) of PLMs via special strategies, e.g., flow-based approach (Li et al., 2020) and whitening method (Huang et al., 2021). Besides, contrastive learning (Wu et al., 2020; Gao et al., 2021) has been used to refine the representations of PLMs. Contrastive Learning. Contrastive learning has been originated in the computer vision (Hadsell et al., 2006; He et al., 2020) and information retrieval (Bian et al., 2021; Zhou et al., 2022) field with significant performance improvement. Usually, it relies on data augmentation strategies such as random cropping and image rotation (Chen et al., 2020; Yan et al., 2021) to produce a set of semantically related positive examples for learning, 6121 and randomly samples negatives from the batch or whole dataset. For sentence representation learning, contrastive learning can achieve a better bal-ance between alignment and uniformity in semantic representation space. Several works further adopt back translation (Fang and Xie, 2020), token shuffling (Yan et al., 2021) and dropout (Gao et al., 2021) to augment positive examples for sentence representation learning. However, the quality of the randomly sampled negatives is seldom studied. Virtual Adversarial Training. Virtual adversarial training (VAT) (Miyato et al., 2019; Kurakin et al., 2017) perturbs a given input with learnable noises to maximize the divergence of the model's prediction with the original label, then utilizes the perturbed examples to improve the generalization (Miyato et al., 2017; Madry et al., 2018). A class of VAT methods can be formulated into solving a min-max problem, which can be achieved by multiple projected gradient ascent steps (Qin et al., 2019). In the NLP field, several studies incorporate adversarial perturbations in the embedding layer, and show its effectiveness on text classification (Miyato et al., 2017), machine translation (Sun et al., 2020), and natural language understanding (Jiang et al., 2020) tasks. This work aims to make use of unlabeled corpus for learning effective sentence representations that can be directly utilized for downstream tasks, e.g., semantic textual similarity task (Agirre et al., 2015). Given a set of input sentences X = { x 1 , x 2 , . . . , x n } , our goal is to learn a representation h i  R d for each sentence x i in an unsupervised manner. For simplicity, we denote this process with a parameterized function h i = f ( x i ) . In this work, we mainly focus on using BERT-based PLMs (Devlin et al., 2019; Liu et al., 2019) to generate sentence representations. Following existing works (Li et al., 2020; Yan et al., 2021), we fine-tune PLMs on the unlabeled corpus via our proposed unsupervised learning approach. After that, for each sentence x i , we encode it by the fine-tuned PLMs and take the representation of the [CLS] token from the last layer as its sentence representation h i . Our proposed framework DCLR focuses on reducing the influence of sampling bias in the contrastive contrastive learning of sentence representations. In this framework, we devise a noise-based negatives generation strategy to reduce the bias caused by the anisotropy PLM-derived representations, and an instance weighting method to reduce the bias caused by false negatives. Concretely, we initialize the noise-based negatives based on a Gaussian distribution and iteratively update these negatives towards non-uniformity maximization. Then, we utilize a complementary model to produce weights for all negatives ( i.e., randomly sampled and the noise-based ones). Finally, we combine the weighted negatives and augmented positive examples for contrastive learning. The overview of our DCLR is presented in Figure 2. We aim to generate new negatives beyond the sentence representation space of PLMs during the training process, to alleviate the sampling bias derived from the anisotropy problem of PLMs (Etha-yarajh, 2019). For each input sentence x i , we first initialize k noise vectors from a Gaussian distribution as the negative representations: {  h 1 ,  h 2 ,    ,  h k }  N (0 ,  2 ) , (1) where  is the standard variance. Since these vectors are randomly initialized from such a Gaussian distribution, they are uniformly distributed within the whole semantic space. By learning to contrast with these new negatives, it is beneficial for the uniformity of sentence representations. To further improve the quality of the new negatives, we consider iteratively updating the negatives to capture the non-uniformity points within the whole semantic space. Inspired by VAT (Miy-ato et al., 2017; Zhu et al., 2020), we design a non-uniformity loss maximization objective to produce gradients for improving these negatives. The non-uniformity loss is denoted as the contrastive loss between the noise-based negatives {  h j } and the positive representations of the original sentence ( h i , h + i ) as: LU ( h i , h + i , {  h } ) =  log e sim ( h i ,h + i ) / u (cid:80)  h j {  h j } e sim ( h i ,  h i ) / u , (2) where  u is a temperature hyper-parameter and sim ( h i , h + i ) is the cosine similarity h  i h + i || h i |||| h + i || . g ( h j ) denotes the gradient of  h j by maximizing the non-uniformity loss between the positive representations and the noise-based negatives. In this way, the noise-based negatives will be optimized towards the non-uniform points of the sentence representation space. By learning to contrast with these negatives, the uniformity of the representation space can be further improved, which is essential for effective sentence representations. In addition to the above noise-based negatives, we also follow existing works (Yan et al., 2021; Gao et al., 2021) that adopt other in-batch representations as negatives {  h  } . However, as discussed before, the sampled negatives may contain examples that have similar semantics with the positive example ( i.e., false negatives). To alleviate this problem, we propose an instance weighting method to punish the false negatives. Since we cannot obtain the true labels or semantic similarities, we utilize a complementary model to produce the weights for each negative. In this paper, we adopt the state-of-the-art SimCSE (Gao et al., 2021) as the complementary model. 1 Given a negative representation h  from {  h  } or {  h } and the representation of the original sentence h i , we utilize the complementary model to produce the 1 For convenience, we utilize SimCSE on BERT-base or RoBERTa-base model as the complementary model. where  is a hyper-parameter of the instance weighting threshold, and sim C ( h i , h  ) is the cosine similarity score evaluated by the complementary model. In this way, the negative that has a higher semantic similarity with the representation of the original sentence will be regarded as a false negative and will be punished by assigning the weight 0. Based on the weights, we optimize the sentence representations with a debiased cross-entropy contrastive learning loss function as L =  log e sim ( h i ,h + i ) / (cid:80) h  {  h }{  h  }  h   e sim ( h i ,h  ) / , (6) where  is a temperature hyper-parameter. In our framework, we follow SimCSE (Gao et al., 2021) that utilizes dropout to augment positive examples h + i . Actually, we can utilize various positive augmentation strategies, and will investigate it in Section 6.1. Our framework DCLR contains three major steps. In the first step, we generate noise-based negatives to extend in-batch negatives. Concretely, we first initialize a set of new negatives via random Gaussian noises using Eq. 1. Then, we incorporate a gradient-based algorithm to adjust the noise-based negatives by maximizing the non-uniform objective using Eq. 3. After several iterations, we can 6123 obtain the noise-based negatives that correspond to the nonuniform points within the whole semantic space, and we mix up them with in-batch negatives to compose the negative set. In the second step, we adopt a complementary model ( i.e., SimCSE) to compute the semantic similarity between the original sentence and each example from the negative set, and produce the weights using Eq. 5. Finally, we augment the positive examples via dropout and utilize the negatives with corresponding weights for contrastive learning using Eq. 6. 4.3.2 Discussion As mentioned above, our approach aims to reduce the influence of the sampling bias about the negatives, and is agnostic to various positive data augmentation methods ( e.g., token cutoff and dropout). Compared with traditional contrastive learning methods (Yan et al., 2021; Gao et al., 2021), our proposed DCLR expands the negative set by introducing noise-based negatives {  h } , and adds a weight term  h  to punish false negatives. Since the noise-based negatives are initialized from a Gaussian distribution and do not correspond to real sentences, they are highly confident negatives to broaden the representation space. By learning to contrast with them, the learning of the contrastive objective will not be limited by the anisotropy representations derived from PLMs. As a result, the sentence representations can span a broader semantic space, and the uniformity of the representation semantic space can be improved. Besides, our instance weighting method also alleviates the false negative problem caused by the randomly sampling strategy. With the help of a complementary model, the false negatives with similar semantics as the original sentence will be detected and punished. Following previous works (Kim et al., 2021; Gao et al., 2021), we conduct experiments on seven standard STS tasks. For all these tasks, we use the SentEval toolkit (Conneau and Kiela, 2018) for evaluation. Semantic Textual Similarity Task. We evaluate our approach on 7 STS tasks: STS 20122016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). These datasets contain pairs of two sentences, whose similarity scores are labeled from 0 to 5. The relevance between gold annotations and the scores predicted by sentence representations is measured by the Spearman correlation. Following the suggestions from previous works (Gao et al., 2021; Reimers and Gurevych, 2019), we directly compute the cosine similarity between sentence embeddings for all STS tasks. Baseline Methods. We compare DCLR with competitive unsupervised sentence representation learning methods, consisting of non-BERT and BERT-based methods: (1) GloVe (Pennington et al., 2014) averages GloVe embeddings of words as the sentence representation. (2) USE (Cer et al., 2018) utilizes a Transformer model that learns the objective of reconstructing the surrounding sentences within a passage. (3) CLS , Mean and First-Last AVG (Devlin et al., 2019) adopt the [CLS] embedding, mean pooling of token representations, average representations of the first and last layers as sentence representations, respectively. (4) Flow (Li et al., 2020) applies mean pooling on the layer representations and maps the outputs to the Gaussian space as sentence representations. (5) Whitening (Su et al., 2021) uses the whitening operation to refine representations and reduce dimensionality. (6) Contrastive (BT) (Fang and Xie, 2020) uses contrastive learning with back-translation for data augmentation to enhance sentence representations. (7) ConSERT (Yan et al., 2021) explores various text augmentation strategies for contrastive learning of sentence representations. (8) SG-OPT (Kim et al., 2021) proposes a contrastive learning method with a self-guidance mechanism for improving the sentence embeddings of PLMs. (9) SimCSE (Gao et al., 2021) proposes a simple contrastive learning framework that utilizes dropout for data augmentation. Implementation Details. We implement our model based on Huggingface's transformers (Wolf et al., 2020). For BERT-base and RoBERTa-base, we start from the pre-trained checkpoints of their original papers. For BERT-large and RoBERTa-large, we utilize the checkpoints of SimCSE for stabilizing the convergence process. Following Sim-6124 Models STS12 STS13 STS14 STS15 STS16 STS-B SICK-R Avg. CSE (Gao et al., 2021), we use 1,000,000 sentences randomly sampled from Wikipedia as the training corpus. During training, we train our models for 3 epoch with temperature  = 0 . 05 using an Adam optimizer (Kingma and Ba, 2015). For BERT-base and RoBERTa-base, the batch size is 128, the learning rate is 3e-5. For BERT-large and RoBERTa-large, the batch size is 256, the learning rate is 3e-5 and 1e-5, respectively. For the four backbone models, we set the instance weighting threshold  as 0.9, 0.85, 0.9 and 0.85, respectively. For each batch, we generate k  batch _ size noise-based negatives as the shared negatives of all instance within it, and k is 1, 2.5, 4 and 5 for BERT-base, RoBERTa-base, BERT-large and RoBERTa-large, respectively. The standard variance of the noise-based negatives is 1, and we update the noise-based negatives four times with the learning rate of 1e-3. We evaluate the model every 150 steps on the development set of STS-B and SICK-R and keep the best checkpoint for evaluation on test sets. To verify the effectiveness of our framework on PLMs, we select BERT-base and RoBERTa-base as the base model. Table 1 shows the results of different methods on seven STS tasks. Based on the results, we can find that the non-BERT methods ( i.e., GloVe and USE) mostly outperform native PLM representation based baselines 6125 ( i.e., CLS, Mean and First-Last AVG). The reason is that directly utilizing the PLM native representations is prone to be influenced by the anisotropy issue. Among non-BERT methods, USE outperforms Glove. A potential reason is that USE encodes the sentence using the Transformer model, which is more effective than simply averaging GloVe embeddings. For other PLM-based approaches, first, we can see that flow and whitening achieve similar results and outperform the native representations based methods by a margin. These two methods adopt specific improvement strategies to refine the representations of PLMs. Second, approaches based on contrastive learning outperform the other baselines in most cases. Contrastive learning can enhance both the alignment between semantically related positive pairs and the uniformity of the representation space using negative samples, resulting in better sentence representations. Furthermore, SimCSE performs the best among all the baselines. It indicates that dropout is a more effective positive augmentation method than others since it rarely hurts the semantics of the sentence. Finally, DCLR performs better than all the baselines in most settings, including the approaches based on contrastive learning. Since these methods mostly utilize randomly sampled negatives ( e.g., in-batch negatives) to learn the uniformity of all sentence representations, it may lead to sampling bias, such as false negatives and anisotropy representations. Different from these methods, our framework adopts an instance weighting method to punish false negatives and a gradient-based algorithm to generate noise-based negatives towards the nonuniform points. In this way, the sampling bias problem can be alleviated, and our model can better learn the uniformity to improve the quality of the sentence representations. In this section, we continue to study the effectiveness of our proposed DCLR. Since our proposed DCLR is a general framework that mainly focuses on negative sampling for contrastive learning of unsupervised sentence representations, it can be applied to other methods that rely on different positive data augmentation strategies. Thus, in this part, we conduct experiments to examine whether our framework can bring improvements with the following positive data augmentation strategies: (1) Token Shuffling that randomly shuffles the order of the tokens in the input sequences; (2) Feature/Token/Span Cutoff (Yan et al., 2021) that randomly erases features/tokens/token spans in the input; (3) Dropout that is similar to SimCSE (Gao et al., 2021). Note that we only revise the negative sampling strategies to implement these variants of our DCLR. As shown in Figure 3, our DCLR can boost the performance of all these augmentation strategies, it demonstrates the effectiveness of our framework with various augmentation strategies. Furthermore, the Dropout strategy leads to the best performance among all the variants. It indicates that dropout is a more effective approach to augment high-quality positives, and is also more appropriate for our approach. Our proposed DCLR incorporates an instance weighting method to punish false negatives and also utilizes noise-based negatives to improve the uniformity of the whole sentence representation space. To verify their effectiveness, we conduct an ablation study for each of the two components on seven STS tasks and report the average value 6126 0 2000 4000 6000 8000 10000 12000 Training steps 3.0 2.8 2.6 2.4 2.2 2.0 U n i f o r m i t y l o ss OursSimCSE Figure 4: The uniformity loss of DCLR and SimCSE using BERT-base on the validation set of STS-B during training. of the Spearman's correlation metric. As shown in Table 2, removing each component would lead to the performance degradation. It indicates that the instance weighting method and the noise-based negatives are both important in our framework. Besides, removing the instance weighting method results in a larger performance drop. The reason may be that the false negatives have a larger effect on sentence representation learning. Besides, we prepare three variants for further comparison: (1) Random Noise directly generates noise-based negatives without the gradient-based optimization; (2) Knowledge Distillation (Hinton et al., 2015) utilizes SimCSE as the teacher model to distill knowledge into the student model during training; (3) Self Instance Weighting adopts the model itself as the complementary model to generate the weights. From Table 2, we can see that these variations don't perform as well as the original DCLR. These results indicate the proposed designs in Section 4 are more suitable for our DCLR framework. Uniformity is a desirable characteristic for sentence representations, describing how well the representations are uniformly distributed. To validate the improvement of the uniformity of our framework, we compare the uniformity loss curves of DCLR and SimCSE using BERT-base during training. where p data is the distribution of all sentence representations, and a smaller value of this loss indicates a better uniformity. As shown in Figure 4, the 100% 30% 10% 3% 1% 0.3% Training data proportion 0 10 20 30 40 50 60 70 80 A v er ag e s p e a r m a n c o rre l a t i o n STS-BSICK-R Figure 5: Performance tuning of our DCLR w.r.t. different amounts of training data. uniformity loss of DCLR is much lower than that of SimCSE in almost the whole training process. Furthermore, we can see that the uniformity loss of DCLR decreases faster as training goes, while the one of SimCSE shows no significant decreasing trend. It might be because our DCLR samples noise-based negatives beyond the representation space, which can better improve the uniformity of sentence representations. To validate the reliability and the robustness of DCLR under the data scarcity scenarios, we conduct few-shot experiments using BERT-base as the backbone model. We train our model via different amounts of available training data from 100% to the extremely small size ( i.e., 0.3%). We report the results evaluated on STS-B and SICK-R tasks. As shown in Figure 5, our approach achieves stable results under different proportions of the training data. Under the most extreme setting with 0.3% data proportion, the performance of our model drops by only 9 and 4 percent on STS-B and SICK-R, respectively. The results reveal the robustness and effectiveness of our approach under the data scarcity scenarios. Such characteristics are important in real-world application. For hyper-parameters analysis, we study the impact of instance weighting threshold  and the proportion of noise-based negatives k . The  is the threshold to punish false negatives, and k is the ratio of the noise-based negatives to the batch size. Both hyper-parameters are important in our framework. Concretely, we evaluate our model with varying values of  and k on the STS-B and SICK-R tasks using the BERT-base model. Weighting threshold. Figure 6(a) shows the influence of the instance weighting threshold  . For the STS-B tasks,  has a significant effect on the model performance. Too large or too small  may lead to a performance drop. The reason is that a larger threshold cannot achieve effective punishment and a smaller one may cause misjudgment of true negatives. In contrast, the SICK-R is insensitive to the changes of  . The reason may be that the problem of false negatives is not serious in this task. Negative proportion. As shown in Figure 6(b), our DCLR performs better when the number of noise-based negatives is close to the batch size. Under these circumstances, the noise-based negatives are more capable to enhance the uniformity of the whole semantic space without hurting the alignment, which is key why DCLR works well. In this paper, we proposed DCLR, a debiased contrastive learning framework for unsupervised sentence representation learning. Our core idea is to alleviate the sampling bias caused by the random negative sampling strategy. To achieve it, in our framework, we incorporated an instance weighting method to punish false negatives during training and generated noise-based negatives to alleviate the influence of anisotropy PLM-derived representation. Experimental results on seven STS tasks have shown that our approach outperforms several competitive baselines. In the future, we will explore other approaches to reducing the bias in contrastive learning of sentence representations ( e.g., debiased pre-training). Besides, we will also consider to apply our method for multilingual or multimodal representation learning. In this section, we discuss the ethical considerations of this work from the following two aspects. First, for intellectual property protection, the code, data and pre-trained models adopted from previous works are granted for research-purpose usage. Second, since PLMs have been shown to capture certain biases from the data they have been pre-trained on (Bender et al., 2021), there is a potential problem about biases that are from the use of PLMs in our approach. There are increasing efforts to address this problem in the community (Ross et al., 2020). This work was partially supported by Beijing Natural Science Foundation under Grant No. 4222027, and National Natural Science Foundation of China under Grant No. 61872369, Beijing Outstanding Young Scientist Program under Grant No. BJJWZYJH012019100020098, the Outstanding Innovative Talents Cultivation Funded Programs 2021 and Public Computing Cloud, Renmin University of China. This work is also supported by Beijing Academy of Artificial Intelligence (BAAI). Xin Zhao is the corresponding author.
abstain abstain abstain abstain objective method result other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective method method objective objective objective objective method abstain abstain objective objective result other other other other other other other other other other other other other other other other other abstain abstain abstain method abstain method method abstain method method method method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain abstain method method method method result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result result abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result result objective method method abstain abstain abstain other other other other