67.9
68.3
72.6
73.1

Summ. MTEB Avg
24.3
25.8
26.5
26.0

55.5
58.9
57.2
60.3

Fine-tuning Datasets GTR models are ﬁne-tuned with “MS-MARCO + NQ”, while Sentence-T5
models use NLI instead. In Table 6, we can see that the “MS-MARCO + NQ” setting performs best
on retrieval tasks, and the NLI data is beneﬁcial for STS and linear probing classiﬁcation. Similar
observations are also made by Muennighoff et al. [40]. Combining all of them leads to the best
overall scores on the MTEB benchmark. This also illustrates the importance of dataset diversity for
learning text embeddings.

Table 7: Data ﬁltering. For the top 2 rows, we train with 1M random text pairs.

# of pairs
1M

All

NFCorpus NQ FiQA Quora DBPedia

w/o ﬁlter
w/ ﬁlter
w/o ﬁlter
w/ ﬁlter

23.0
26.8
34.5
35.8

15.1
22.7
35.4
39.0

18.5
24.5
39.1
40.0

83.1
85.0
85.7
85.7

18.2
27.5
32.9
35.4

Scifact Avg
34.9
51.4
40.7
57.5
50.0
72.5
51.6
73.7
E5