2.1 Pile-CC

Common Crawl is a collection of website crawls
from 2008 onwards, including raw web pages,
metadata and text extractions. Due to the raw na-
ture of the dataset, Common Crawl has the ad-
vantage of including text from diverse domains,
but at the cost of varying quality data. Due to
this, use of Common Crawl typically necessi-
tates well-designed extraction and ﬁltering. Our
Common Crawl-based dataset, Pile-CC, uses jus-
Text (Endrédy and Novák, 2013) on Web Archive
ﬁles (raw HTTP responses including page HTML)
for extraction, which yields higher quality output
than directly using the WET ﬁles (extracted plain-
text).
The Pile- An 800GB Dataset of Diverse Text for Language Modeling