Harvesting semi-structured data sources Large-scale high-quality datasets like C4 [48] and
CCMatrix [51] are vital for the success of language model pre-training and machine translation. For
learning text embeddings, existing works either utilize small-scale human-annotated data such as NLI
[22] and MS-MARCO [8] or adopt heuristics such as random cropping [28] to obtain large-scale
but very noisy supervision signals.
Instead, we curate a text pair dataset CCPairs (Colossal Clean text Pairs) by harvesting heterogeneous
semi-structured data sources. Let (q, p) denote a text pair consisting of a query q and a passage p.
Here we use “passage” to denote word sequences of arbitrary length, which can be a short sentence, a
paragraph, or a long document. Our dataset includes (post, comment) pairs from Reddit 3, (question,
upvoted answer) pairs from Stackexchange 4, (entity name + section title, passage) pairs from English
E5