Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. Word and sentence similarity tasks have become the de facto evaluation method. It leads models to overfit to such evaluations, negatively impacting embedding models' development. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. Further, we propose a new intrinsic evaluation method called EvalRank , which shows a much stronger correlation with downstream tasks. Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments. Finally, the practical evaluation toolkit is released for future benchmarking purposes. 1 1 Introduction Distributed representation of words (Bengio et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) and sentences (Kiros et al., 2015; Conneau et al., 2017; Reimers and Gurevych, 2019; Gao et al., 2021) have shown to be extremely useful in transfer learning to many NLP tasks. Therefore, it plays an essential role in how we evaluate the quality of embedding models. Among many evaluation methods, the word and sentence similarity task gradually becomes the de facto intrinsic evaluation method. Figure 1 shows examples from word and sentence similarity datasets. In general, the datasets consist of pairs of words ( w 1 , w 2 ) (or sentences) and human-annotated similarity scores S h . To evaluate an embedding model  (  ) , we first extract embeddings for ( w 1 , w 2 ): ( e 1 , e 2 ) = (  ( w 1 ) ,  ( w 2 ) ). Then, a similarity measure is applied to compute an predicted score S p = sim ( e 1 , e 2 ) , where cosine similarity is adopted as sim unquestionably in the majority of cases. Finally, the correlation between S h and S p is computed, and a higher correlation suggests good alignment with human annotations and a better embedding model. Many studies, especially those targeting on information retrieval via semantic search and clustering (Reimers and Gurevych, 2019; Su et al., 2021), have used the similarity task as the only or main evaluate method (Tissier et al., 2017; Mu et al., 2018; Arora et al., 2017; Li et al., 2020; Gao et al., 2021). We observe a number of issues in word or sentence similarity tasks ranging from dataset collection to the evaluation paradigm, and consider that focusing too much on similarity tasks would negatively impact the development of future embedding models. The significant concerns are summarized as follows, which generally apply to both word and sentence similarity tasks. First, the definition of similarity is too vague. There exist complicated relationships between sampled data pairs, and almost all relations contribute to the similarity score, which is challenging to non-expert annotators. Second, the similarity evaluation tasks are not directly 6060 relevant to the downstream tasks. We believe it is because of the data discrepancy between them, and the properties evaluated by similarity tasks are not the ones important to downstream applications. Third, the evaluation paradigm can be tricked with simple post-processing methods, making it unfair to benchmark different models. Inspired by Spreading-Activation Theory (Collins and Loftus, 1975), we propose to evaluate embedding models as a retrieval task, and name it as EvalRank to address the above issues. While similarity tasks measure the distance between similarity pairs from all similarity levels, EvalRank only considers highly similar pairs from a local perspective. Our main contributions can be summarized as follows: 1 We point out three significant problems for using word and sentence similarity tasks as the de facto evaluation method through analysis or experimental verification. The study provides valuable insights into embeddings evaluation methods. 2 We propose a new intrinsic evaluation method, EvalRank , that aligns better with the properties required by various downstream tasks. 3 We conduct extensive experiments with 60+ models and 10 downstream tasks to certify the effectiveness of our evaluation method. The practical evaluation toolkit is released for future benchmarking purposes. Word embedding has been studied extensively, and popular work (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) are mainly built on the distributional hypothesis (Harris, 1954), where words that appear in the same context tend to share similar meanings. The early work on sentence embedding are either built upon word embedding (Arora et al., 2017; Rckl et al., 2018; Almarwani et al., 2019) or follow the distributional hypothesis on a sentence level (Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018). Recent development of sentence embedding are incorporating quite different techniques including multi-task learning (Cer et al., 2018), supervised inference data (Conneau et al., 2017; Reimers and Gurevych, 2019), contrastive learning (Zhang et al., 2020; Carlsson et al., 2020; Yan et al., 2021; Gao et al., 2021) and pre-trained language models (Li et al., 2020; Wang and Kuo, 2020; Su et al., 2021). Nonetheless, even though different methods choose different evaluation tasks, similarity task is usually the shared task for benchmarking purposes. Similarity task is originally proposed to mimic human's perception about the similarity level between word or sentence pairs. The first word similarity dataset was collected in 1965 (Rubenstein and Goodenough, 1965), which consists of 65 word pairs with human annotations. It has been a standard evaluation paradigm to use cosine similarity between vectors for computing the correlation with human judges (Agirre et al., 2009). Many studies raise concerns about such evaluation paradigm. Faruqui et al. (2016) and Wang et al. (2019b) points out some problems with word similarity tasks, including low correlation with downstream tasks and lack of task-specific similarity. Reimers et al. (2016), Eger et al. (2019) and Zhelezniak et al. (2019) states current evaluation paradigm for Semantic Textual Similarity (STS) tasks are not ideal. One most recent work (Abdalla et al., 2021) questions about the data collection process of STS datasets and creates a new semantic relatedness dataset (STR) by comparative annotations (Lou-viere and Woodworth, 1991). There are also other intrinsic evaluation methods for word and sentence embedding evaluation, but eventually did not gain much popularity. Word analogy task is first proposed in (Mikolov et al., 2013a,c) to detect linguistic relations between pairs of word vectors. Zhu and de Melo (2020) recently expanded the analogy concept to sentence level. However, the analogy task is more heuristic and fragile as an evaluation method (Gladkova et al., 2016; Rogers et al., 2017). Recently, probing tasks have been proposed to measure intriguing properties of sentence embedding models without worrying much about practical applications (Zhu et al., 2018; Conneau et al., 2018; Baranckov and Bo-jar, 2019). Because of the lack of effective intrinsic evaluation methods, Reimers and Gurevych (2019) and Wang et al. (2021) seeks to include more domain-specific tasks for evaluation. In this work, we discuss the problems of similarity tasks both on word and sentence levels. They are highly similar from data collection to evaluation 6061 paradigm and are troubled by the same problems. First, the concept of similarity and relatedness are not well-defined. Similar pairs are related but not vise versa. Taking synonym, hypernym, and antonym relations as examples, the similarity rank should be synonym > hypernym > antonym while the relatedness rank should be synonym > hypernym  antonym. This was not taken into consideration when constructing datasets. Agirre et al. (2009) intentionally split one word similarity dataset into similarity and relatedness subsets. However, we find that obtained subsets are erroneous towards polysemy, and the relatedness between pair (stock', egg', 1.81) is much lower than pair (stock', oil', 6.34). It is because only the financial stock market' is compared but not the stock of supermarkets. Furthermore, relationships between samples are far more complicated than currently considered, which is a challenge to all current datasets. Second, the annotation process is not intuitive to humans. The initial goal of the similarity task is to let the model mimic human perception. However, we found that the instructions on similarity levels are not well defined. For example, on STS 13  16 datasets, annotators must label sentences that share some details' with a score of 2 and on the same topic' with a score of 1. According to priming effect theory, (Meyer and Schvan-eveldt, 1971; Weingarten et al., 2016), humans are more familiar with ranking several candidate samples based on one pivot sample (priming stimulus). Therefore, a more ideal way of annotation is to give one pivot sample (e.g. cup') and rank candidates with different similarity levels (e.g. trophy', tableware', food', article', cucumber'). In other words, it is more intuitive for human to compare (a,b) > (a,c) than (a,b) > (c,d) as far as similarity is concerned. However, in practice, it is hard to collect a set of candidates for each pivot sample, especially for sentences. In previous studies, it was found that the performance of similarity tasks shows little or negative correlation with the performance of downstream tasks (Faruqui et al., 2016; Wang et al., 2019b, 2021). An illustration is shown in Table 1a. We think there are two reasons behind 1) low testing Score (rank) STS-B SST2 MR GloVe 47.95 (4) 79.52 (6  ) 77.54 (5  ) InferSent 70.94 (3) 83.91 (3) 77.61 (4  ) BERT-cls 20.29 (6) 86.99 (1  ) 80.99 (1  ) BERT-avg 47.29 (5) 85.17 (2  ) 80.05 (2  ) BERT-flow 71.76 (2) 80.67 (4  ) 77.01 (6  ) BERT-whitening 71.79 (1) 80.23 (5  ) 77.96 (3  ) First, similarity datasets have their data source and are not necessarily close to the corpus of downstream tasks. For example, Baker et al. (2014) collect word pairs for verbs only while Luong et al. (2013) intentionally test on rare words. Also, for STS datasets, (Agirre et al., 2012) annotates on sentence pairs from paraphrases, video captions, and machine translations, which has limited overlap on downstream tasks like sentiment classification. Second, the original goal for the similarity task is to mimic human perceptions. For example, STS datasets are originally proposed as a competition to find the most effective STS systems instead of a gold standard for generic sentence embedding evaluation. Some properties evaluated by similarity tasks are trivial to downstream tasks, and it is more important to test on mutually important ones. As examples in Figure 1, the similarity tasks inherently require the model to predict sim( p 1 ) > sim( p 2 ) and sim( p 5 ) > sim( p 4 ), which we believe are unnecessary for most downstream applications. Instead, similar pairs are more important than less similar pairs for downstream applications (Keklinen, 2005; Reimers et al., 2016). Therefore, it is enough for good embedding models to focus on gathering similar pairs together while keeping dissimilar ones far away to a certain threshold. As similarity tasks become one de facto evaluation method for embedding models, recent work tend to overfit the current evaluation paradigm, including the choice of similarity measure and the post-processing step. Similarity Metrics. Cosine similarity is the default choice for similarity tasks. However, simply changing the similarity metric to other commonly used ones can lead to contradictory results. In Table 1b, we compare recent five BERT-based sentence embedding models including BERT (De-vlin et al., 2019), BERT-whitening (Su et al., 2021), BERT-flow (Li et al., 2020), SBERT (Reimers and Gurevych, 2019) and SimCSE (Gao et al., 2021). 2 The results on standard STS-Benchmark testset are reported under both cosine and l 2 similarity. As we can see, the performance rank differs under different similarity metrics. This is especially true for BERT-flow and BERT-whitening, which do not even outperform their baseline models when evaluating with l 2 metric. Therefore, we can infer that some models overfit to the default cosine metric for similarity tasks. Whitening Tricks. A number of studies attempted the post-processing of word embeddings (Mu et al., 2018; Wang et al., 2019a; Liu et al., 2019b) and sentence embeddings (Arora et al., 2017; Liu et al., 2019a; Li et al., 2020; Su et al., 2021). The shared concept is to obtain a more isotropic embedding space (samples evenly distributed across directions) and can be summarized as a space whitening process. Even though the whitening tricks help a lot with similarity tasks, we found it is usually not applicable to downstream tasks or even hurt the model performance. 3 We think the whitening methods are overfitted to similarity tasks and would like 2 Experimental details in Appendix B. 3 Analysis in Appendix C.1. First, we take the whole STS-Benchmark dataset and create subsets of sentence pairs from certain similarity levels. We test on two baseline sentence embedding models: GloVe, BERT; three whitening tricks: ABTT on GloVe (Mu et al., 2018), BERT-whitening, BERT-flow; two strong sentence embedding models that perform well on both STS and downstream tasks: SBERT, SimCSE. Figure 2 shows the result, and we can see that the whitening-based methods are boosting the baseline performance mainly for less similar pairs (e.g., pairs with a similarity score within [2,0]). In contrast, the models that perform well on downstream tasks show consistent improvement on all subsets with different similarity scores. As discussed in Section 3.2, highly similar pairs are more critical than less similar pairs for downstream tasks. Since the postprocessing methods mainly help with less similar pairs, they do not help much on downstream tasks. In cognitive psychology, Spreading-Activation Theory (SAT) (Collins and Loftus, 1975; Anderson, 1983) is to explain how concepts store and interact within the human brain. Figure 3 shows one example about the concept network. In the network, only highly related concepts are connected. To find the relatedness between concepts like engine and street , the activation is spreading through mediating concepts like car and ambulance with decaying factors. Under this theory, the similarity task is measuring the association between any two concepts in the network, which requires complicated long-distance activation propagation. Instead, to test the soundness of the concept network, it is enough to ensure the local connectivity between concepts. Moreover, the long-distance relationships can be inferred thereby with various spreading activation 6063 Type # pos pairs # background samples Source EvalRank Word 5,514 22,207 Word Similarity Datasets & Wiki Sent 6,989 24,957 STS-Benchmark & STR Table 2: Statistics of EvalRank Datasets algorithms (Cohen and Kjeldsen, 1987). Therefore, we propose EvalRank to test only on highly related pairs and make sure they are topologically close in the embedding space. It also alleviates the problems of similarity tasks. First, instead of distinguishing multifaceted relationships, we only focus on highly related pairs, which are intuitive to human annotators. Second, it shows a much stronger correlation with downstream tasks as desired properties are measured. Third, as we treat the embedding space from a local perspective, it is less affected by the whitening methods. We frame the evaluation of embeddings as a retrieval task. To this purpose, the dataset of EvalRank contains two sets: 1) the positive pair set P = { p 1 , p 2 , ..., p m } and 2) the background sample set C = { c 1 , c 2 , ..., c n } . Each positive pair p i = ( c x , c y ) in P consists of two samples in C that are semantically similar. For each sample ( c x ) and its positive correspondence ( c y ), a good embedding model should has their embeddings ( e x , e y ) close in the embedding space. Meantime, the other background samples should locate farther away from the sample c x . Some samples in the background may also be positive samples. We assume it barely happens and is negligible if good datasets are constructed. Formally, given an embedding model  (  ) , the embeddings for all samples in C are computed as { e 1 , e 2 , ..., e n } = {  ( c 1 ) ,  ( c 2 ) , ...,  ( c n ) } . The cos similarity and l 2 similarity between two samples ( c x , c y ) are defined as: S cos ( c x , c y ) = e Tx e y || e x ||  || e y || S l 2 ( c x , c y ) = 1 1 + || e x  e y || Further, the similarity score is used to sort all background samples in descending order and the performance at each positive pair p i is measured by the rank of c x 's positive correspondence c y w.r.t all background samples: rank i = rank ( S ( c x , c y ) , [ || nj =1 ,j (cid:54) = x S ( c x , c j )]) where || refers to the concatenation operation. To measure the overall performance of model  (  ) on all positive pairs in P , the mean reciprocal rank (MRR) and Hits@k scores are reported and a higher score indicates a better embedding model: MRR = 1 m m (cid:88) i =1 1 rank i Hits @ k = 1 m m (cid:88) i =1 1 [ rank i  k ] Note that there are two similarity metrics, and we found that S cos shows a better correlation with downstream tasks while S l 2 is more robust to whitening methods. We use S cos in the experiments unless otherwise specified. Word-Level. We collect the positive pairs from 13 word similarity datasets (Wang et al., 2019b). For each dataset, the pairs with the highest 25% similarity score are gathered as positive pairs. Background word samples contain all words that appear in the similarity datasets. Further, we augment the background word samples using the most frequent 20,000 words from Wikipedia corpus. Sentence-Level. Similarly, the pairs with top 25% similarity/relatedness score from STS-Benchmark dataset (Cer et al., 2017) and STR dataset (Abdalla et al., 2021) are collected as positive pairs. All sentences that appear at least once are used as the background sentence samples. In both cases, if positive pair ( c x , c y ) exists, the reversed pair ( c y , c x ) is also added as positive pairs. Detailed statistics of EvalRank datasets are listed in Table 2. 4.4 Alignment and Uniformity Recently, Wang and Isola (2020) identifies the alignment and uniformity properties as an explanation to the success of contrastive loss. It shares many similarities with our method and can also shed light on why EvalRank works. First, the alignment property requires similar samples to have similar features, which aligns with the objective of 6064 SCICITE MR CR MPQA SUBJ SST2 SST5 TREC MRPC SICK-E WS-353-All 62.87 43.68 40.94 37.50 15.57 41.65 45.03 34.70 8.98 57.96 WS-353-Rel 66.13 47.92 45.15 41.77 11.65 47.25 48.18 26.36 20.56 61.83 WS-353-Sim 67.86 45.94 43.97 38.68 17.41 44.03 50.32 34.85 10.67 56.13 RW-STANFORD 75.56 74.65 55.35 66.08 46.82 81.50 68.25 45.91 13.08 43.29 MEN-TR-3K 66.91 44.15 45.37 39.14 1.70 38.51 42.11 22.82 28.63 71.26 MTURK-287 68.48 65.95 48.01 52.36 31.94 71.96 58.01 29.22 7.54 36.23 MTURK-771 79.93 60.87 49.45 57.92 24.04 62.75 62.03 29.14 17.44 60.23 SIMLEX-999 68.20 48.02 40.90 46.43 19.03 47.30 50.95 38.14 15.32 60.26 SIMVERB-3500 65.13 45.60 36.95 47.04 21.57 45.16 48.56 41.74 10.70 58.08 EvalRank MRR 89.96 87.91 68.23 78.03 51.35 91.54 83.36 48.15 25.70 61.34 Hits@1 85.91 83.69 66.93 81.43 55.95 89.74 79.46 43.53 28.82 53.86 Hits@3 90.11 88.82 69.92 82.05 54.52 93.32 84.41 48.44 30.87 62.77 Table 3: Spearman's rank correlation (   100 ) between performance scores of word-level intrinsic evaluation and downstream tasks, where the best is marked with bold and second best with underline. EvalRank . Second, the uniformity property is measured by the average Gaussian distance between any two samples. In contrast, EvalRank focuses on the distance between points from a local perspective and would require the pivot sample to have longer distances to any background samples than its positive candidate. Measuring the distance from a local perspective has unique advantages because the learned embedding space will likely form a manifold and can only approximates euclidean space locally. Therefore, simple similarity metrics like cos or l 2 are not suitable to model long-distance relationships. A good intrinsic evaluator can test the properties that semantically similar samples are close in vector space (Reimers and Gurevych, 2019; Gao et al., 2021) and serve as prompt information to real-world applications. As EvalRank directly test on the first property, we design experiments to show the correlation with various downstream tasks as a comparison of intrinsic evaluators. To be comprehensive, we first collect as many embedding models as possible and test them on the intrinsic evaluator and downstream task. The Spearman's rank correlation is computed between the results, and a higher score indicates better correlation with downstream tasks and better intrinsic evaluator. Meantime, we do not think similarity evaluations should be discarded, even though it fails to correlate well with downstream applications. It has its advantages as aiming to mimic human perception about semantic-related pairs. Word Embedding Models. We collect 19 word embedding models from GloVe (Pennington et al., 2014), word2vec (Mikolov et al., 2013b), fastText (Bojanowski et al., 2017), Dict2vec (Tissier et al., 2017) and PSL (Wieting et al., 2015). Meantime, we apply ABTT (Mu et al., 2018) post-processing to all models to double the total number of embedding models. When testing on downstream tasks, the simplest bag-of-words feature is used as sentence representations in order to focus on measuring the quality of word embeddings. Word Similarity Tasks. 9 word similarity datasets are compared as the baseline methods including WS-353-All (Finkelstein et al., 2001), WS-353-Rel (Agirre et al., 2009), WS-353-Sim (Agirre et al., 2009), RW-STANFORD (Luong et al., 2013), MEN-TR-3K (Bruni et al., 2014), MTURK-287 (Radinsky et al., 2011), MTURK-771 (Halawi et al., 2012), SIMLEX-999 (Hill et al., 2015), SIMVERB-3500 (Gerz et al., 2016). The word similarity datasets with less than 200 pairs are not selected to avoid evaluation occasionality. Cosine similarity and Spearman's rank correlation are deployed for all similarity tasks. Downstream Tasks. SentEval (Conneau and Kiela, 2018) is a popular toolkit in evaluating sentence embeddings. We use 9 downstream tasks from SentEval including MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), MPQA (Wiebe et al., 2005), SUBJ (Pang and Lee, 2004), SST2 (Socher et al., 2013), SST5 (Socher et al., 2013), TREC (Li and Roth, 2002), MRPC (Dolan et al., 2004), SICK-E (Marelli et al., 2014). Previous work spot 6065 SCICITE MR SST2 EvalRank 89.96 87.91 91.54 w/o wiki vocabs 88.55 83.99 88.26 w/ WN synonym 90.56 86.56 91.12 w/ l 2 metric 77.47 78.34 81.51 Table 4: Ablation study on variants of EvalRank . that SentEval tasks are biased towards sentiment analysis (Wang et al., 2018). Therefore, we add one extra domain-specific classification task SCICITE (Cohan et al., 2019) which assigns intent labels (background information, method, result comparison) to sentences collected from scientific papers that cite other papers. For all tasks, a logistic regression classifier is used with cross-validation to predict the class labels. Table 3 shows the word-level results. In short, EvalRank outperforms all word similarity datasets with a clear margin. For evaluation metrics, we can see that Hits@3 score shows a higher correlation than MRR and Hits@1 scores. However, the gap between the evaluation metrics is not big, which makes them all good measures. Among all 10 downstream tasks, EvalRank shows a strong correlation (  >0.6) with 7 tasks and a very strong correlation (  >0.8) with 5 tasks. While, among all word similarity datasets, only one dataset (RW-STANFORD) shows a strong correlation with one downstream task (SST2). For word similarity datasets, RW-STANFORD dataset shows the best correlation with downstream tasks. It confirms the finding in Wang et al. (2019b) that this dataset contains more high-quality and low-frequency word pairs. Ablation Study. We experiment with several variants of our EvalRank method and the result is shown in Table 4. First, if we do not augment the background word samples with the most frequent 20,000 words from the Wikipedia corpus, it leads to certain performance downgrading. Without suf-ficient background samples, positive pairs are not challenging enough to test each model's capability. Second, we tried to add more positive samples (e.g. 5k samples) using synonym relations from WordNet (WN) database (Miller, 1998). However, no obvious improvement is witnessed because the EvalRank MRR Hits@1 Hits@3 GloVe 13.15 4.66 15.72 word2vec 12.88 4.57 14.35 fastText 17.22 5.77 19.99 Dict2vec 12.71 4.03 13.04 synonym pairs in WN contain too many noisy pairs. Last, for similarity measures, we notice that cos similarity is consistently better than l 2 similarity while both outperform word similarity baselines. Benchmarking Results. In Table 5a, we compared four popular word embedding models, including GloVe, word2vec, fastText, and Dict2vec, where fastText achieves the best performance. Sentence Embedding Models. We collect 67 embedding models, where 38 of them are built upon word embeddings with bag-of-words features and 29 of them are neural-network-based models. For neural-network-based models, we collect variants from InferSent (Conneau et al., 2017), BERT (De-vlin et al., 2019), RoBERTa (Liu et al., 2020), BERT-flow (Li et al., 2020), BERT-whitening (Su et al., 2021), SBERT (Reimers and Gurevych, 2019) and SimCSE (Gao et al., 2021). Sentence Similarity Tasks. We evaluate on 7 standard semantic textual similarity datasets including STS12  16 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS-Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). Recently, Abdalla et al. (2021) questioned the labeling process of STS datasets and released a new semantic textual relatedness (STR) dataset, which is also included in our experiments. from SentEval evaluation toolkit, including MR, CR, MPQA, SUBJ, SST2, SST5, TREC, as well as the domain-specific classification task SCICITE. We exclude the MRPC and SICK-E because they are highly similar with STS tasks (Conneau and Kiela, 2018). Table 6 shows the sentence-level results. EvalRank outperform all sentence similarity datasets with a clear margin. For evaluation metric, Hits@1 shows a higher correlation comparing with MRR and Hits@3. Among all 7 downstream tasks, EvalRank shows strong correlation (  > 0 . 6 ) with 6 tasks. For sentence similarity datasets, no one clearly outperforms others. Additionally, we found that STR dataset shows the worst correlation with downstream tasks. Even though STR adopts a better data annotation schema than STS datasets, it still fol-SCICITE MR SST2 EvalRank (STS-B + STR) MRR 65.95 83.43 80.97 Hits@1 69.01 85.39 82.65 Hits@3 63.35 83.92 80.36 EvalRank (STS-B) MRR 63.05 75.85 72.87 Hits@1 66.22 77.94 75.20 Hits@3 61.23 75.49 72.92 EvalRank (STR) MRR 63.51 83.28 80.20 Hits@1 66.59 84.53 82.14 Hits@3 60.68 82.55 79.42 Table 7: Performance under different data sources. lows the previous standard evaluation paradigm and is exposed to the same problems. It further ver-ifies our discussion about problems with sentence similarity evaluation. Correlation Visualization. Figure 4 shows the performance rank of 67 sentence embedding models on five tasks, including 2 downstream tasks (MR, SST2) and 3 intrinsic evaluations (STS-B, STR, EvalRank ). The models' performance rank on the MR task is used as the pivot. As MR and SST2 datasets are both related to sentiment analysis, they correlate well with each other. Among the three intrinsic evaluation tasks, EvalRank shows a higher correlation with downstream tasks as the blue dots roughly follow the trend of red dots. In contrast, the dots of STS-B and STR are dispersed in different regions. This shows that the performance of STS-B and STR is not a good indicator of the performance on downstream tasks. mance of EvalRank with different data sources. By combining the positive pairs collected from both STS-B and STR datasets, EvalRank leads to the best performance. Interestingly, according to our results, even though STR evaluation does not correlate well with downstream tasks, the positive pairs collected from STR have better quality than STS-B. It also confirms the argument that STR improves the dataset collection process (Abdalla et al., 2021). Benchmarking Results. Table 5b benchmarked seven popular sentence embedding models. As the widely accepted SOTA model, SimCSE outperforms others with a clear margin. In this work, we first discuss the problems with current word and sentence similarity evaluations and proposed EvalRank , an effective intrinsic evaluation method for word and sentence embedding models. It shows a higher correlation with downstream tasks. We believe that our evaluation method can have a broader impact in developing future embedding evaluation methods, including but not limited to its multilingual and task-specific extensions. This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project No. A18A2b0046) and Science and Engineering Research Council, Agency of Science, Technology and Research (A*STAR), Singapore, through the National Robotics Program under Human-Robot Interaction Phase 1 (Grant No. 192 25 00054).
abstain abstain abstain abstain objective objective method abstain other method abstain abstain abstain objective abstain abstain abstain result abstain abstain abstain abstain method abstain objective abstain objective objective objective method objective other other other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain abstain abstain method abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain method method abstain method method abstain method abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain objective abstain objective other