Learning high-quality sentence representations is a fundamental problem of natural language processing which could benefit a wide range of downstream tasks. Though the BERT-like pre-trained language models have achieved great success, using their sentence representations directly often results in poor performance on the semantic textual similarity task. Recently, several contrastive learning methods have been proposed for learning sentence representations and have shown promising results. However, most of them focus on the constitution of positive and negative representation pairs and pay little attention to the training objective like NT-Xent, which is not sufficient enough to acquire the discriminating power and is unable to model the partial order of semantics between sentences. So in this paper, we propose a new method ArcCSE, with training objectives designed to enhance the pairwise discriminative power and model the entailment relation of triplet sentences. We conduct extensive experiments which demonstrate that our approach outperforms the previous state-of-the-art on diverse sentence related tasks, including STS and SentEval. Learning sentence representations, which encodes sentences into fixed-sized dense vectors such that semantically similar ones stay close, is a fundamental problem of natural language processing. It could benefit a wide range of downstream applications such as information retrieval, semantic similarity comparison, question answering, and so on. Recently, with the great success of pre-trained Transformer-based language models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020; Liu et al., 2019) like BERT, they have been widely adopted for generating sentence representations. A straightforward way is by leveraging the [CLS] embedding (Devlin et al., 2019)  1 0 1 or applying mean pooling on the last layers of a BERT-like pre-trained language model (Reimers and Gurevych, 2019). However, the sentence embeddings coming from a pre-trained language model without further fine-tuning could not capture the semantic meaning of sentences very well as shown in Figure 1(a), and sometimes even un-derperform non-contextualized embeddings like GloVe (Pennington et al., 2014). To make pre-trained language models more suitable for generating sentence embeddings, supervised methods like SBERT (Reimers and Gurevych, 2019) are proposed, which improve the performance by fine-tuning on a labeled dataset. As labeled data is not available or expensive to annotate in many tasks or domains, it is of great value for developing unsupervised/self-supervised approaches for learning sentence representations. So recent works like BERT-Flow (Li et al., 2020) and BERT-Whitening (Su et al., 2021) propose postprocessing methods to improve the BERT-based sentence representation. They address that the non-smooth anisotropic semantic space of BERT is a bottleneck and alleviate the problem through normalizing flows and whitening operation. To further improve the quality of sentence representations, several works (Kim et al., 2021; Yan et al., 2021; Giorgi et al., 2021; Carlsson et al., 2021; Gao et al., 2021) adopt self-supervised contrastive learning approach, which learns sentence representations by minimizing the distance of positive sentence representation pairs and maximizing the distance of negative pairs. In these works, positive pairs are often constituted through data augmentation or encoders with different structure or parameters, while negative pairs are derived from different sentences within the same batch. Then contrastive learning objective like normalized temperature-scaled cross-entropy loss (NT-Xent) (Chen et al., 2020; Gao et al., 2021) is used for optimizing. A typical example unsup-SimCSE (Gao et al., 2021) achieves state-of-the-art performance with a simple and effective idea of using standard dropout for data augmentation. Though existing contrastive methods for learning sentence representation have shown promising results, most of them focus on the positive and negative pairs constitution, and the optimization objective itself is not fully exploited. The contrastive learning objective NT-Xent loss used in recent works (Yan et al., 2021; Giorgi et al., 2021; Gao et al., 2021) is a variation of cross-entropy loss with softmax function. Recent studies (Wang et al., 2018; Deng et al., 2019) have shown that the traditional softmax-based loss is insufficient to acquire the discriminating power, as shown in Figure 1(b) in which SimCSE-BERT base adopts the NT-Xent loss and could not separate s b and s c completely. In addition, the current optimization objectives only models sentence relations in a pairwise perspective, which tries to pull sentences with similar semantics closer and push dissimilar ones away from each other. However, there are different degrees of semantic similarity among related sentences. For example in Figure 1(d), s b is more similar to s a than s c is. The current optimization objectives lack the ability to model the partial order of semantics between sentences. To alleviate these problems, in this paper, we propose a new approach ArcCSE for sentence representation learning. For pairwise sentence relation modeling, we propose Additive Angular Margin Contrastive Loss (ArcCon Loss), which enhances the pairwise discriminative power by maximizing the decision margin in the angular space. Besides, in order to model the partial order of semantics between sentences, we propose a new self-supervised task that captures the entailment relation among triplet sentences. The task is implemented through automatically constituted triplet sentences with entailment relation among them. A visualization example of the generated representations through ArcCSE is shown in Figure 1(c). We evaluate our method on standard semantic textual similarity (STS) tasks and SentEval transfer tasks, and it outperforms the previous state-of-the-art approaches. Early works usually learn sentence representations by augmenting the idea of word2vec (Mikolov et al., 2013), such as predicting surrounding sentences (Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) or summing up n-gram embeddings (Pagliardini et al., 2018). With the rise of pre-trained language models, many works try to generate sentence representations through BERT-like models. A common way is leveraging the [CLS] embedding or applying mean pooling on the last layers of BERT (Reimers and Gurevych, 2019; Li et al., 2020). Instead of using BERT embeddings directly, BERT-Flow (Li et al., 2020) and BERT-Whitening (Su et al., 2021) further improve sentence representation through post-processing. Recently, several works adopt the contrastive learning framework for sentence representation learning. They propose different strategies to constitute contrastive pairs, either through different data transforming methods (Zhang et al., 2020; Yan et al., 2021; Giorgi et al., 2021), or through encoders with different structures or parameters (Carlsson et al., 2021; Kim et al., 2021; Gao et al., 2021). A typical example SimCSE (Gao et al., 2021) uses dropout as data augmentation strategy and achieves state-of-the-art performance. However, most existing works pay little attention to the training objective and use the traditional contrastive BERT-like Encoder (Dropout Turn on) BERT-like Encoder (Dropout Turn off) Parameter Sharing Al Jaber's first long distance travel was of 800km which he covered by circling Qatar. loss directly, which is insufficient in discrimination and unable to model the partial order of semantics between sentences. So, in our work, we propose a new approach that jointly models the pairwise and triple-wise sentence relations and further improves the sentence representations' quality. The goal of Deep Metric Learning (DML) is to learn a function that maps objects into an embedded space, in which similar objects stay close and dissimilar ones are far away. In order to achieve this goal, many approaches have been proposed, and designing appropriate loss functions plays a key role in it. Contrastive training objectives like Contrastive Loss (Chopra et al., 2005), N-Pair Loss (Sohn, 2016), Structured Loss (Song et al., 2016) and Triplet Margin Loss (Ma et al., 2021) apply the definition of metric learning directly. These objectives are among the earliest training objectives used for deep metric learning. Later, softmax-based losses which learn a center for each class and penalize the distances between deep features and their corresponding class centers achieve more promising results in supervised metric learning. Typical examples like Center Loss (Wen et al., 2016), SphereFace (Liu et al., 2017), CosFace (Wang et al., 2018) and ArcFace (Deng et al., 2019) are widely adopted in deep learning applications such as face recognition and sentence classification (Coria et al., 2020). However, these losses need class labels and are not suitable for learning sentence representations. So inspired by ArcFace, we propose a new training objective ArcCon that does not need class labels and can model pairwise sentence relations with more discriminative power than traditional contrastive training objectives. In this section, we present ArcCSE, an angular based contrastive sentence representation learning framework, which could generate superior sentence embeddings from unlabeled data. Given a pre-trained language model M and an unlabeled text dataset D , the task is fine-tuning M on D so that the sentence representations generated through M could be more semantic discriminative. model pairwise and triple-wise sentence relations simultaneously, as shown in Figure 2. We start with angular margin based contrastive learning in Section 3.1, which models pairwise relations between sentences by pulling semantic similar ones closer while pushing dissimilar ones away. Then we introduce the method which models the partial order of semantics between automatically constituted triplet sentences in Section 3.2. To model the positive/negative pairwise relations between sentences, we first need to generate sentence representations and group them into positive and negative pairs. Then we feed these pairs to a training objective for optimizing. Given a collection of sentences D = { s i } Ni =1 , we generate the sentence representations through a BERT-like pre-trained language model M . Following SimCSE, we use dropout as the data augmentation method. For each sentence s i , we generate two different representations h i and h  i from s i by passing s i to M twice with independently sampled dropout masks. These two representations with the same semantics constitute a positive pair, while the negative pairs are derived from the representations of different sentences within the same batch. After getting the positive and negative sentence pairs, we put them into a training objective for model fine-tune. The most widely adopted training objective is NT-Xent loss (Chen et al., 2020; Gao et al., 2021), which has been used in previous sentence and image representation learning methods and can be formulated as follows: L NT-Xent =  log e sim ( h i ,h  i ) / (cid:80) n j =1 e sim ( h i ,h j ) / (1) where sim ( h i , h j ) is the cosine similarity h T i h j || h i |||| h j || ,  is a temperature hyperparameter and n is the number of sentences within a batch. Though the training objective tries to pull representations with similar semantics closer and push dissimilar ones away from each other, these representations may still not be sufficiently discriminative and not very robust to noise. Let us denote angular  i,j as follows:  i,j = arccos (cid:18) h T i h j || h i ||  || h j || (cid:19) (2) NT-Xent B e tt e r A li gn m e n t ArcCon D ec i s i on M a r g i n OptimizationDirection Better Uniformity B e tt e r A li gn m e n t Better Uniformity Figure 3: Comparison of NT-Xent loss and ArcCon loss. The decision boundary for h i in NT-Xent is  i,i  =  i,j , as show in Figure 3. Due to lack of decision margin, a small perturbation around the decision boundary may lead to an incorrect decision. To overcome the problem, we propose a new training objective for sentence representation learning by adding an additive angular margin m between positive pair h i and h  i . We named it Additive Angular Margin Contrastive Loss (ArcCon Loss), which can be formulated as follows: L arc =  log e cos (  i,i  + m ) / e cos (  i,i  + m ) / + (cid:80) j  = i e cos(  j,i ) / (3) In this loss, the decision boundary for h i is  i,i  + m =  i,j , as show in Figure 3. Compared with NT-Xent, it further pushed h i towards to the area where  i,i  get smaller and  i,j get larger, by increasing the compactness of sentence representations with the same semantics and enlarging the discrepancy of different semantic representations. This help enhance the alignment and uniformity properties (Wang and Isola, 2020), which are two key measures of representation quality related to contrastive learning, indicating how close between positive pair embeddings and how well the embeddings are uniformly distributed. The quantitative analysis is illustrated in Section 4.5. Besides, the decision boundary leaves an extra margin m to boundary  i,i  =  i,j which is often used during inference, making it more tolerant to noise and more robust. All these properties make ArcCon loss more discriminative than traditional training objectives like NT-Xent. Compared with Arcface (Deng et al., 2019) which is often used in large-scale fine-grained categorization in computer vision community, ArcCon loss does not need classification labels, and could handle contrastive task properly. Previously the training objectives for sentence representation learning like NT-Xent loss only considered pairwise sentence relations, in which sentences are either similar or dissimilar in semantics. But in fact, there are varying degrees of semantic similarity. For example, sentence s 2 could be more similar to sentence s 1 than sentence s 3 to s 1 . Existing methods lack the ability to model such partial order of semantics between sentences. In order to distinguish the slight differences in semantics between different sentences, we propose a new self-supervised task which models the entailment relation of automatically generated triplet sentences. For each sentence s i in the text dataset D , we first generate an external sentence s  i by masking contiguous segments of s i with a masking rate of 20%. Then we enlarge the masking area and get a new sentence s  i with a masking rate of 40% to s i . The masking rates are set up experimentally, and an ablation study about the effect of masking rates is illustrated in Section 4.4. An example of the masking procedure is shown as follows: s i Al Jaber's first long distance travel was of 800km which he covered by circling Qatar. s  i Al Jaber's first long distance travel was of 800km which he covered by circling Qatar. s  i Al Jaber's first long distance travel was of 800km which he covered by circling Qatar. We can constitute a triplet ( s i , s  i , s  i ) with entailment relation among them. Though in rare cases, the strategy may generate sentences that do not exhibit the desired relationship and introduce some noise, the entailment relation holds true most of the time. We expect encountering enough data will reinforce the correct ones whereas the impact of incorrect ones will diminish. Since the s i , s  i and s  i are similar literally and semantically, generating their representations with dropout noise may obscure their entailment relation and add inaccurate signals to the representation learning process. So we turn off the dropout of the encoder when modeling the triplet relation. As s  i is more similar to s i in semantics than s  i is, we could model such relation with a triplet objective: L tri = max (cid:0) 0 , sim ( h i ,  h  i )  sim ( h i ,  h  i ) + m (cid:1) (4) in which  h i is the sentence representation of s i generated without dropout noise and sim ( i, j ) is the cosine similarity between i and j . As the semantic difference between s  i and s  i may be subtle depending on the original sentence s i and the masked words, here we set m to zero. Combine formula (3) and formula (4), the final form of our training objective is: L = L arc +  L tri (5) in which  is a coefficient. Evaluation Tasks We evaluate our method on two kinds of sentence related tasks: Unsupervised Semantic Textual Similarity (STS): These tasks measure the model's ability to estimate the semantic similarities between sentences. SentEval Transfer Tasks: These tasks measure the effectiveness of sentence embeddings used in downstream transfer tasks. Baselines We compare ArcCSE to several representative methods on STS and SentEval tasks, such as average GloVe embeddings (Pennington et al., 2014), Skip-thought (Kiros et al., 2015), average BERT embeddings from the last layer (De-vlin et al., 2019), BERT-Flow (Li et al., 2020), and BERT-Whitening (Su et al., 2021). We also include the recently proposed contrastive learning methods, such as ISBERT (Zhang et al., 2020), CT-BERT (Carlsson et al., 2021), ConSERT (Yan et al., 2021), and the current state-of-the-art method SimCSE (Gao et al., 2021). Implementation Details We train ArcCSE with the pre-trained checkpoints of BERT base and BERT large (Devlin et al., 2019). We also employ our method to SBERT (Reimers and Gurevych, 2019), which has been trained on NLI datasets, to verify the generalizability of our method. Following SimCSE (Gao et al., 2021), we use the output of the MLP layer on top of the [CLS] as Method STS12 STS13 STS14 STS15 STS16 STS-B SICK-R Avg. the sentence representation during training, and use the [CLS] output without MLP layer for evaluation. The dropout rate is set to 0.1. For ArcCon loss, we set the angular margin m to 10 degrees and the temperature  to 0.05. When modeling the entailment relation of triplet sentences, we set the masking ratios as 20% and 40% respectively. Since the semantic difference between triplet sentences is more obvious for long sentences, we filter out sentences with less than 25 words and use the left ones for the triplet loss. The loss coefficient  is set to 0.1 experimentally. We use one million random sampled sentences from English Wikipedia for training, which has been used in previous work (Gao et al., 2021) 1 . During training, the sentences are sampled by length. We set different maximum sentence lengths for ArcCon loss and triplet loss to save memory. The length is set to 32 for the ArcCon loss in large models, and to the maximum length within a batch for all other cases. We train our model for one epoch and the learning rate is set to 3e-5 for base 1 https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt models and 1e-5 for large models. We search the batch size within {8, 16, 32} and always update the parameters every 64 steps. The model is optimized by the AdamW with Sharpness-Aware Minimization (Foret et al., 2021) and default configurations. We evaluate our model every 125 training steps on the development set of STS-B, and the best checkpoint is used for the final evaluation on test sets. Our implementation is based on Hugging-Face's Transformers (Wolf et al., 2020). We conduct experiments on 7 semantic textual similarity (STS) tasks, including STS tasks 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017), and SICK-Relatedness (Marelli et al., 2014). Within these datasets, each sample contains two sentences and a gold score between 0 and 5 which indicates their semantic similarity. We use SentEval toolkit (Con-neau and Kiela, 2018) for evaluation and report the Spearman's correlation following previous works (Reimers and Gurevych, 2019; Gao et al., 2021). from which we can see that ArcCSE outperforms the previous approaches. Compared with the previous state-of-the-art method SimCSE, ArcCSE-BERT base raises the average Spearman's correlation from 76.25% to 78.11%, and ArcCSE-BERT large further pushes the results to 79.37%. The performance is even better than strong supervised method SBERT, which has already been trained on NLI datasets. Furthermore, we can also employ our method to SBERT and improve its performance to 79.06% and 80.69% for the base and large models respectively, which is more effective than SimCSE. We also explore the improvements made by the ArcCon loss and triplet loss independently based on BERT base . From Table 1 we can see that with ArcCon loss alone, the average Spearman's correlation is 77.25%. When combining the traditional NT-Xent loss with our proposed triplet loss, the average Spearman's correlation is 77.02%. Both of them outperform the previous state-of-the-art method SimCSE, whose average Spearman's correlation is 76.25%. This demonstrates the effectiveness of ArcCon and triplet loss we proposed. We evaluate our model with SentEval toolkit on several supervised transfer tasks, including: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). For each task, SentEval trains a logistic regression classifier on top of the sentence embeddings and tests the performance on the downstream task. For a fair comparison, we do not include models with auxiliary tasks like masked language modeling. The results are shown in Table 2. We can see that ArcCSE performs on par or better than baseline methods in both BERT base and BERT large level. This demonstrates the effectiveness of our method in learning domain-specific sentence embeddings. Effect of Angular Margin The angular margin m in ArcCon loss affects the discriminative power directly. To investigate the effect of m , we conduct an experiment by varying m from 0 degrees to 20 degrees, increased by 2 degrees at each step. We tune the hyper-parameter based on Spearman's correlation on the development set of STS-B following previous works (Kim et al., 2021; Gao et al., 2021). The results are shown in Figure 4. Figure 4: Effect of the angular margin m in ArcCon loss. We can see that the best performance is achieved when m = 10 , either larger or smaller margin degrade the performance. This matches our intuition since small m may have little effect, and large m may negatively influence the positive pair relation modeling. Effect of Temperature The temperature  in ArcCon Loss affects its effectiveness, so we carry out an experiment with  varying from 0.01 to 0.1, increased by 0.01 at each step. The results are shown in Figure 5. We can see that the model ArcCSE-BERT base STS12 STS13 STS14 STS15 STS16 STS-B SICK-R Avg. Effect of Masking Ratios The masking ratios determine the sentences generated for the entailment relation modeling and their differences in semantics, so we conduct an experiment to explore the effect of different masking ratios. The first masking ratio r 1 is varied from 10% to 25%, increased by 5% for each step. The second masking ratio r 2 is derived by adding an extra value r d to r 1 . r d is varied from 10% to 35%, increased by 5% for each step. The results are shown in Figure 6. Figure 6: Effect of the masking ratios. We can see that large differences between the two masking ratios tend to lead lower Spearman's correlation compared to the smaller ones. The reason may be that the larger the semantic difference is, the easier for the model to estimate the entailment relations among the triplet sentences, which makes the triplet loss less helpful. The best performance is achieved when r 1 is 20% and r 2 is 40%, and the corresponding Spearman's correlation is 0.847. We use them as our hyper-parameters. Effect of on-off Switching of Dropout The on-off switching of dropout in the BERT-like sentence encoder affects the generated sentence representations directly. Since dropout performs a kind of averaging over the ensemble of possible subnetworks, an embedding generated with dropout turned off can be seen as a kind of "averaging" representation, while an embedding generated with dropout turned on can be seen as generated through a subnetwork. In ArcCSE, we use the embeddings generated with the encoder dropout turned on as input for ArcCon loss, which regularizes the network by making representations generated through different subnetworks similar. When modeling the entailment relation, we generate "averaging" representations with dropout turn-off to avoid inaccurate signals. In order to verify our intuition, we conduct two experiments with different dropout settings. In the first experiment, we feed ArcCon two sentence representations generated with dropout turns on and off respectively. We carry out this experiment with angular margins ranging between 2 degrees to 12 degrees and report the best result. In the second one, we feed the triplet loss representations that are generated with dropout turns on and maintain the other settings. The results are shown in Table 3. We can see that the original settings that turn dropout on for ArcCon and turn dropout off for triplet loss achieve the best performance, which confirms our intuition. Effect of Coefficient in the Training Objective The coefficient  in the final optimization objective adjusts the relative weights between ArcCon and the triplet loss, as shown in formula (5). To find the most suitable  , we conduct an experiment by varying  from 0 to 1.2 and increased by 0.1 at each step. The results are shown in Figure 7. We can see that the best performance is achieved Figure 7: Effect of the coefficient in the training objective. when  = 0 . 1 , and the corresponding Spearman's correlation is 0.847. This demonstrates that we can get the best performance by combining ArcCon and the triplet loss with proper  . Alignment and uniformity are two properties closely related to contrastive learning and could be used to measure the quality of representa-tions(Wang and Isola, 2020). Alignment favors encoders that generate similar representations for similar instances. It could be defined with the expected distance between embeddings of the positive paired instances:  align = E ( x,x + )  p pos (cid:13)(cid:13) f ( x )  f (cid:0) x + (cid:1)(cid:13)(cid:13) 2 (6) where p pos denotes the distribution of positive paired instances. Uniformity prefers uniformly distributed representations, which helps preserve maximal information. It could be defined as:  uniform = log E x,y i.i.d  p data e  2  f ( x )  f ( y )  2 (7) where p data denotes whole data distribution. To justify the inner workings of our approach, we calculate the alignment and uniformity metrics every 10 steps during training on the STS-B development set. We compare our approach with SimCSE and visualize the results in Figure 8. We can see that compared to the original BERT checkpoint, both ArcCSE and SimCSE improve the alignment and uniformity measures during training. ArcCSE performs better on the alignment measure and on par with SimCSE on the uniformity measure. This verifies the intuition of our approach and demonstrates that ArcCSE could help improve the quality of sentence representations. In this work, we propose ArcCSE, a self-supervised contrastive learning framework for learning sen-(a)  align tence representation. We propose a new optimizing objective ArcCon loss to model pairwise sentence relations with enhanced discriminating power, and a new self-supervised task to model the partial order of semantics between sentences. Experimental results on semantic textual similarity tasks (STS) and SentEval tasks demonstrate that both techniques bring substantial improvements and our method outperforms previous state-of-the-art method for sentence representation learning.
abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain abstain result other other other other other other other other other objective other other other other other other other objective method abstain abstain method method method method method method method abstain method other abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method abstain method abstain method method method method method abstain abstain method method method method method abstain abstain method method method abstain method abstain method abstain other method abstain method method abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain objective abstain abstain objective objective