  Semantic similarity between natural language texts is typically measured
either by looking at the overlap between subsequences (e.g., BLEU) or by using
embeddings (e.g., BERTScore, S-BERT). Within this paper, we argue that when we
are only interested in measuring the semantic similarity, it is better to
directly predict the similarity using a fine-tuned model for such a task. Using
a fine-tuned model for the STS-B from the GLUE benchmark, we define the
STSScore approach and show that the resulting similarity is better aligned with
our expectations on a robust semantic similarity measure than other approaches.

Computation and Language;Machine Learning