What you are trying to do is clearly one of the GLUE tasks 4:

3.2 SIMILARITY AND PARAPHRASE TASKS
MRPC The Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether
the sentences in the pair are semantically equivalent. Because the classes are imbalanced (68%
positive), we follow common practice and report both accuracy and F1 score.
QQP The Quora Question Pairs2 dataset is a collection of question pairs from the community
question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. As in MRPC, the class distribution in QQP is unbalanced (63% negative), so we
report both accuracy and F1 score. We use the standard test set, for which we obtained private labels
from the authors. We observe that the test set has a different label distribution than the training set.
STS-B The Semantic Textual Similarity Benchmark (Cer et al., 2017) is a collection of sentence
pairs drawn from news headlines, video and image captions, and natural language inference data.
Each pair is human-annotated with a similarity score from 1 to 5; the task is to predict these scores.
Follow common practice, we evaluate using Pearson and Spearman correlation coefficients.

What I suggest you to do, is to follow the following tutorial 17 to pre-train your model on the dataset that is the most similar to what you are trying to do (ex: GLUE, QQP instead of GLUE MRCP in the tutorial)
There is even available Leaderboard where you can find which model perform the best on QQP.