discarded pages not classiﬁed as references.
C4 [15%]. During exploratory experiments, we
observed that using diverse pre-processed Com-
monCrawl datasets improves performance. We thus
included the publicly available C4 dataset (Raffel
et al., 2020) in our data. The preprocessing of C4
also contains deduplication and language identiﬁ-
cation steps: the main difference with CCNet is
the quality ﬁltering, which mostly relies on heuris-
tics such as presence of punctuation marks or the
number of words and sentences in a webpage.
Github [4.5%]. We use the public GitHub
dataset available on Google BigQuery. We only
kept projects that are distributed under the Apache,
BSD and MIT licenses. Additionally, we ﬁltered
low quality ﬁles with heuristics based on the line
length or proportion of alphanumeric characters,
and removed boilerplate, such as headers, with reg-
ular expressions. Finally, we deduplicate the result-
ing dataset at the ﬁle level, with exact matches.
LLaMA- Open and Efficient Foundation Language Models