  In this paper, we adopted a retrospective approach to examine and compare
five existing popular sentence encoders, i.e., Sentence-BERT, Universal
Sentence Encoder (USE), LASER, InferSent, and Doc2vec, in terms of their
performance on downstream tasks versus their capability to capture basic
semantic properties. Initially, we evaluated all five sentence encoders on the
popular SentEval benchmark and found that multiple sentence encoders perform
quite well on a variety of popular downstream tasks. However, being unable to
find a single winner in all cases, we designed further experiments to gain a
deeper understanding of their behavior. Specifically, we proposed four semantic
evaluation criteria, i.e., Paraphrasing, Synonym Replacement, Antonym
Replacement, and Sentence Jumbling, and evaluated the same five sentence
encoders using these criteria. We found that the Sentence-Bert and USE models
pass the paraphrasing criterion, with SBERT being the superior between the two.
LASER dominates in the case of the synonym replacement criterion.
Interestingly, all the sentence encoders failed the antonym replacement and
jumbling criteria. These results suggest that although these popular sentence
encoders perform quite well on the SentEval benchmark, they still struggle to
capture some basic semantic properties, thus, posing a daunting dilemma in NLP
research.

Computation and Language