We address the task of unsupervised Semantic Textual Similarity (STS) by ensembling diverse pre-trained sentence encoders into sentence meta-embeddings . We apply, extend and evaluate different meta-embedding methods from the word embedding literature at the sentence level, including dimensionality reduction (Yin and Sch  utze, 2016), generalized Canonical Correlation Analysis (Rastogi et al., 2015) and cross-view auto-encoders (Bolle-gala and Bao, 2018). Our sentence meta-embeddings set a new unsupervised State of The Art (SoTA) on the STS Benchmark and on the STS12STS16 datasets, with gains of between 3.7% and 6.4% Pearson's r over single-source systems. Word meta-embeddings have been shown to exceed single-source word embeddings on word-level semantic benchmarks (Yin and Schutze, 2016; Bollegala and Bao, 2018). Presumably, this is because they combine the complementary strengths of their components. There has been recent interest in pre-trained uni-versal sentence encoders, i.e., functions that encode diverse semantic features of sentences into fixed-size vectors (Conneau et al., 2017). Since these sentence encoders differ in terms of their architecture and training data, we hypothesize that their strengths are also complementary and that they can benefit from meta-embeddings. To test this hypothesis, we adapt different meta-embedding methods from the word embedding literature. These include dimensionality reduction (Yin and Schutze, 2016), cross-view autoencoders (Bollegala and Bao, 2018) and Generalized Canonical Correlation Analysis (GCCA) (Rastogi et al., 2015). The latter method was also used by Poerner and Schutze (2019) for domain-specific Duplicate Question Detection. Our sentence encoder ensemble includes three models from the recent literature: Sentence-BERT (Reimers and Gurevych, 2019), the Universal Sentence Encoder (Cer et al., 2017) and averaged ParaNMT vectors (Wieting and Gimpel, 2018). Our meta-embeddings outperform every one of their constituent single-source embeddings on STS1216 (Agirre et al., 2016) and on the STS Benchmark (Cer et al., 2017). Crucially, since our meta-embeddings are agnostic to the contents of their ensemble, future improvements may be possible by adding new encoders. Word embeddings are functions that map word types to vectors. They are typically trained on unlabeled corpora and capture word semantics (e.g., Mikolov et al. (2013); Pennington et al. (2014)). Word meta-embeddings combine ensembles of word embeddings by various operations: Yin and Schutze (2016) use concatenation, SVD and linear projection, Coates and Bollegala (2018) show that averaging word embeddings has properties similar to concatenation. Rastogi et al. (2015) apply generalized canonical correlation analysis (GCCA) to an ensemble of word vectors. Bollegala and Bao (2018) learn word meta-embeddings using autoencoder architectures. Neill and Bollegala (2018) evaluate different loss functions for autoencoder word meta-embeddings, while Bollegala et al. (2018) explore locally linear mappings. Sentence embeddings are methods that produce one vector per sentence. They can be grouped into two categories: F 1 (e.g., SBERT) ... (a) Word embedding average sentence encoders take a (potentially weighted) average of pre-trained word embeddings. Despite their inability to understand word order, they are surprisingly effective on sentence similarity tasks (Arora et al., 2017; Wieting and Gimpel, 2018; Ethayarajh, 2018) (b) Complex contextualized sentence encoders, such as Long Short Term Memory Networks (LSTM) (Hochreiter and Schmidhuber, 1997) or Transformers (Vaswani et al., 2017). Contextualized encoders can be pre-trained as unsupervised language models (Peters et al., 2018; Devlin et al., 2019), but they are usually improved on supervised transfer tasks such as Natural Language Inference (Bowman et al., 2015). Sentence meta-embeddings have been explored less frequently than their word-level counterparts. Kiela et al. (2018) create meta-embeddings by training an LSTM sentence encoder on top of a set of dynamically combined word embeddings. Since this approach requires labeled data, it is not applicable to unsupervised STS. Tang and de Sa (2019) train a Recurrent Neural Network (RNN) and a word embedding average encoder jointly on a large corpus to predict similar representations for neighboring sentences. Their approach trains both encoders from scratch, i.e., it cannot be used to combine existing encoders. Poerner and Schutze (2019) propose a GCCA-based multi-view sentence encoder that combines domain-specific and generic sentence embeddings for unsupervised Duplicate Question Detection. In this paper, we extend their approach by exploring a wider range of meta-embedding methods and an ensemble that is more suited to STS. Semantic Textual Similarity (STS) is the task of rating the similarity of two natural language sentences on a real-valued scale. Related applications are semantic search, duplicate detection and sentence clustering. Supervised SoTA systems for STS typically apply cross-sentence attention (Devlin et al., 2019; Raffel et al., 2019). This means that they do not scale well to many real-world tasks. Supervised siamese models (Reimers and Gurevych, 2019) on the other hand, while not competitive with cross-sentence attention, can cache sentence embeddings independently of one another. For instance, to calculate the pairwise similarities of N sentences, a cross-sentence attention system must calculate O ( N 2 ) slow sentence pair embeddings, while the siamese model calculates O ( N ) slow sentence embeddings and O ( N 2 ) fast vector similarities. Below, we assume access to an ensemble of pre-trained sentence encoders, denoted F 1 . . . FJ . Every F j maps from the (infinite) set of possible sentences S to a fixed-size d j -dimensional vector. Word meta-embeddings are usually learned from a finite vocabulary of word types (Yin and Schutze, 2016). Sentence embeddings lack such a vocabu-lary, as they can encode any member of S . Therefore, we train on a sample S  S , i.e., on a corpus of unlabeled sentences. We create naive sentence meta-embeddings by concatenating (Yin and Schutze, 2016) or averaging 1 (Coates and Bollegala, 2018) sentence embeddings. 1 If embeddings have different dimensionalities, we pad the shorter ones with zeros. F j ( s ) = F j ( s ) ||F j ( s ) || 2 Yin and Schutze (2016) use Singular Value Decomposition (SVD) to compactify concatenated word embeddings. The method is straightforward to extend to sentence meta-embeddings. Let X conc  R | S | (cid:80) j d j with x conc n = F conc ( s n )  E s  S [ F conc ( s )] Let USVT  X conc be the d -truncated SVD. F svd ( s (cid:48) ) = VT ( F conc ( s (cid:48) )  E s  S [ F conc ( s )]) Given random vectors x 1 , x 2 , Canonical Correlation Analysis (CCA) finds linear projections such that  T 1 x 1 and  T 2 x 2 are maximally correlated. Generalized CCA (GCCA) extends CCA to three or more random vectors. Bach and Jordan (2002) show that a variant of GCCA reduces to a generalized eigenvalue problem on block matrices:     1 , 1 0 0 0  ... 0 0 0  J,J      1 . . .  J   =   0  ...  1 ,J  ... 0  ...  J, 1  ... 0      1 . . .  J   where  j,j (cid:48) = E s  S [( F j ( s )   j )( F j (cid:48) ( s )   j (cid:48) ) T ]  j = E s  S [ F j ( s )] For stability, we add d j (cid:80) d j n =1 diag(  j,j ) n to diag(  j,j ) , where  is a hyperparameter. We stack the eigenvectors of the topd eigenvalues into matrices  j  R d  d j and define the GCCA meta-embedding of sentence s (cid:48) as: F gcca ( s (cid:48) ) = J (cid:88) j =1  j ( F j ( s (cid:48) )   j ) F gcca corresponds to MV-DASE in Poerner and Schutze (2019). Autoencoder meta-embeddings are trained by gradient descent to minimize some cross-embedding reconstruction loss. For example, Bollegala and Bao (2018) train feed-forward networks (FFN) to encode two sets of word embeddings into a shared space, and then reconstruct them such that mean squared error with the original embeddings is minimized. Neill and Bollegala (2018) evaluate different reconstruction loss functions: Mean Squared Error (MSE), Mean Absolute Error (MAE), KL-Divergence (KLD) or squared cosine distance (1-COS) 2 . We extend their approach to sentence encoders as follows: Every sentence encoder F j has a trainable encoder E j : R d j  R d and a trainable decoder D j : R d  R d j , where d is a hyperparameter. Our training objective is to reconstruct every embedding x j (cid:48) from every E j ( x j ) . This results in J 2 loss terms, which are jointly optimized: L ( x 1 . . . x J ) = (cid:88) j (cid:88) j (cid:48) l ( x j (cid:48) , D j (cid:48) ( E j ( x j ))) where l is one of the reconstruction loss functions listed above. The autoencoder meta-embedding of a new sentence s (cid:48) is: F ae ( s (cid:48) ) = (cid:88) j E j ( F j ( s (cid:48) )) 4 Experiments 4.1 Data We train on all sentences of length < 60 from the first file ( news.en-00001-of-00100 ) of the tok-enized, lowercased Billion Word Corpus (BWC) (Chelba et al., 2014) (  302K sentences). We evaluate on STS12  STS16 (Agirre et al., 2016) and the unsupervised STS Benchmark test set (Cer et al., dimensionality STS12 STS13 STS14 STS15 STS16 STS-B single:ParaNMT d = 600 67.5/66.3 62.7/62.8 77.3/74.9 80.3/80.8 78.3/79.1 79.8/78.9 single:USE d = 512 62.6/63.8 57.3/57.8 69.5/66.0 74.8/77.1 73.7/76.4 76.2/74.6 single:SBERT d = 1024 66.9/66.8 63.2/64.8 74.2/74.3 77.3/78.3 72.8/75.7 76.2/79.2 single:ParaNMT  up-projection  d = 1024 67.3/66.2 62.1/62.4 77.1/74.7 79.7/80.2 77.9/78.7 79.5/78.6 single:USE  up-projection  d = 1024 62.4/63.7 57.0/57.5 69.4/65.9 74.7/77.1 73.6/76.3 76.0/74.5 meta:conc d = 2136 72.7/71.3 68.4/68.6 81.0/79.0 84.1/ 85.5 82.0 / 83.8 82.8/83.4 meta:avg d = 1024 72.5/71.2 68.1/68.3 80.8/78.8 83.7/85.1 81.9/83.6 82.5/83.2 meta:svd d = 1024 71.9/70.8 68.3/68.3 80.6/78.6 83.8/85.1 81.6/83.6 83.4/83.8 meta:gcca (hyperparams on dev set) d = 1024 72.8 / 71.6 69.6 / 69.4 81.7 / 79.5 84.2 / 85.5 81.3/83.3 83.9 / 84.4 meta:ae (hyperparams on dev set) d = 1024 71.5/70.6 68.5/68.4 80.1/78.5 82.5/83.1 80.4/81.9 82.1/83.3 Ethayarajh (2018) (unsupervised) 68.3/-66.1/78.4/-79.0/-/-79.5/Wieting and Gimpel (2018) (unsupervised) 68.0/-62.8/77.5/-80.3/78.3/-79.9/Tang and de Sa (2019) (unsupervised meta) 64.0/-61.7/73.7/-77.2/76.7/-Hassan et al. (2019)  (unsupervised meta) 67.7/-64.6/75.6/-80.3/79.3/-77.7/Poerner and Schutze (2019) (unsupervised meta) -/--/-/--/-/-80.4/Reimers and Gurevych (2019) (sup. siamese SoTA) -/--/-/--/-/--/86.2 Raffel et al. (2019) (supervised SoTA) -/--/-/--/-/-93.1/92.8 Table 2: Results on STS1216 and STS Benchmark test set. 2017). 2 These datasets consist of triples ( s 1 , s 2 , y ) , where s 1 , s 2 are sentences and y is their ground truth semantic similarity. The task is to predict similarity scores  y that correlate well with y . We predict  y = cos( F ( s 1 ) , F ( s 2 )) . Previous work on STS differs with respect to (a) the correlation metric and (b) how to aggregate the sub-testsets of STS1216. To maximize comparability, we report both Pearson's r and Spearman's  . On STS1216, we aggregate by a non-weighted average, which diverges from the original shared tasks (Agirre et al., 2016) but ensures comparability with more recent baselines (Wieting and Gimpel, 2018; Ethayarajh, 2018). Results for individual STS12 16 sub-testsets can be found in the Appendix. We select our ensemble according to the following criteria: Every encoder should have near-SoTA performance on the unsupervised STS benchmark, and the encoders should not be too similar with regards to their training regime. For instance, we do not 2 We use SentEval for evaluation (Conneau and Kiela, 2018). Since original SentEval does not support the unsupervised STS Benchmark, we use a non-standard repository ( https://github.com/sidak/SentEval ). We manually add the missing STS13-SMT subtask. use Ethayarajh (2018), which is a near-SoTA unsupervised method that uses the same word vectors as ParaNMT (see below). We choose the Universal Sentence Encoder (USE) 3 (Cer et al., 2018), which is a Transformer trained on skip-thought, conversation response prediction and Natural Language Inference (NLI), Sentence-BERT (SBERT) 4 (Reimers and Gurevych, 2019), which is a pre-trained BERT transformer finetuned on NLI, and ParaNMT 5 (Wi-eting and Gimpel, 2018), which averages word and 3-gram vectors trained on backtranslated similar sentence pairs. To our knowledge, ParaNMT is the current single-source SoTA on the unsupervised STS Benchmark. We set d = 1024 in all experiments, which corresponds to the size of the biggest single-source embedding (SBERT). The value of  (GCCA), as well as the autoencoder depth and loss function are tuned on the STS Benchmark development set (see 3 https://tfhub.dev/google/ universal-sentence-encoder/2 4 https://github.com/UKPLab/ sentence-transformers . We use the large-nlimean-tokens model, which was not finetuned on STS. 5 https://github.com/jwieting/ para-nmt-50m full without without without ensemble ParaNMT USE SBERT meta:svd 85.0 / 85.4 79.6/81.3 79.7/81.4 83.7/83.5 meta:gcca 85.5 / 86.1 84.9/84.8 83.8/83.8 85.4/85.4 meta:ae 85.1 / 85.5 76.5/80.3 82.5/83.5 28.7/41.0 Table 3: Ablation study: Pearson's r  100 / Spearman's   100 on STS Benchmark development set when one encoder is left out. Table 1). We train the autoencoder for a fixed number of 500 epochs with a batch size of 10,000. We use the Adam optimizer (Kingma and Ba, 2014) with  1 = 0 . 9 ,  2 = 0 . 999 and learning rate 0 . 001 . Our main baselines are our single-source embeddings. Wieting and Kiela (2019) warn that high-dimensional sentence representations can have an advantage over low-dimensional ones, i.e., our meta-embeddings might be better than lower-dimensional single-source embeddings due to size alone. To exclude this possibility, we also up-project smaller embeddings by a random d  d j matrix sampled from: U (  1 (cid:112) d j , 1 (cid:112) d j ) Since the up-projected sentence embeddings perform slightly worse than their originals (see Table 2, rows 45), we are confident that performance gains by our meta-embeddings are due to content rather than size. Table 2 shows that even the worst of our meta-embeddings consistently outperform their single-source components. This underlines the overall usefulness of ensembling sentence encoders, irrespective of the method used. GCCA outperforms the other meta-embeddings on five out of six datasets. We set a new unsupervised SoTA on the unsupervised STS Benchmark test set, reducing the gap with the supervised siamese SoTA of Reimers and Gurevych (2019) from 7% to 2% Spearman's  . Interestingly, the naive meta-embedding methods (concatenation and averaging) are competitive with SVD and the autoencoder, despite not needing any unsupervised training. In the case of concatenation, this comes at the cost of increased dimensionality, which may be problematic for downstream applications. The naive averaging method by Coates and Bollegala (2018) however does not have this problem, while performing only marginally worse than concatenation. Table 3 shows that all single-source embeddings contribute positively to the meta-embeddings, which supports their hypothesized complementarity. This result also suggests that further improvements may be possible by extending the ensemble. All of our meta-embeddings are fast to train, either because they have closed-form solutions (GCCA and SVD) or because they are lightweight feed-forward nets (autoencoder). The underlying sentence encoders are more complex and slow, but since we do not update them, we can apply them to the unlabeled training data once and then reuse the results as needed. As noted in Section 2.4, cross-sentence attention systems do not scale well to many real-world STS-type tasks, as they do not allow individual sentence embeddings to be cached. Like Reimers and Gurevych (2019), our meta-embeddings do not have this problem. This should make them more suitable for tasks like sentence clustering or real-time semantic search. Inspired by the success of word meta-embeddings, we have shown how to apply different meta-embedding techniques to ensembles of sentence encoders. All sentence meta-embeddings consistently outperform their individual single-source components on the STS Benchmark and the STS1216 datasets, with a new unsupervised SoTA set by our GCCA meta-embeddings. Because sentence meta-embeddings are agnostic to the size and specifics of their ensemble, it should be possible to add new encoders to the ensemble, potentially improving performance further. Acknowledgments. This work was supported by Siemens AG and by the European Research Council (# 740516).
method abstain objective abstain abstain abstain objective method abstain abstain abstain result objective other other other other other other other other other other other other other other other other other other objective other other other other other other method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other method abstain method abstain method other method method abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result objective abstain abstain abstain