We do not believe that document level ﬁltering
is sufﬁcient for WET ﬁles because many of the
issues with WET ﬁles stem from intra-document
boilerplate. We also ﬁnd many of the heuristics
used in Raffel et al. (2019), such as the removal
of all lines without terminal punctuation, the word
"javascript", and 3-sentence deduplication to be too
aggressive.
C.1.2 Extraction
In addition to jusText, we also considered Traﬁ-
latura, Newspaper, Goose3, and DragNet. While
we were originally intending on creating an extrac-
tion benchmark, this proved infeasible given our
available resources, and we chose jusText based on
visual inspection of the output. In inspection, we
noticed that jusText has the characteristic that it dis-
cards more data than many other extractors, which
is not a major drawback given the large volume
of CC data available. This was as expected, given
The Pile- An 800GB Dataset of Diverse Text for Language Modeling