Conneau et al., 2017, show in the InferSent-Paper (Supervised Learning of Universal Sentence Representations from Natural Language Inference Data) that training on Natural Language Inference (NLI) data can produce universal sentence embeddings.
The datasets labeled sentence pairs with the labels entail, contradict, and neutral. For both sentences, we compute a sentence embedding. These two embeddings are concatenated and passed to softmax classifier to derive the final label.
As shown, this produces sentence embeddings that can be used for various use cases like clustering or semantic search.
Datasets
We train the models on the SNLI and on the MultiNLI dataset. We call the combination of the two datasets AllNLI.
For a training example, see examples/training_nli_bert.py. 
Pretrained models
We provide the various pre-trained models. The performance was evaluated on the test set of the STS benchmark dataset using Spearman rank correlation.
