4. The investigation and documentation of this
dataset, which we hope will better inform re-
searchers about how to use it as well as moti-
vate them to undertake similar investigations
of their own data.

2 The Pile Datasets

The Pile is composed of 22 constituent sub-datasets,
as shown in Table 1. Following Brown et al. (2020),
we increase the weights of higher quality compo-
nents, with certain high-quality datasets such as
Wikipedia being seen up to 3 times (“epochs”) for

2

Component
Pile-CC
PubMed Central
Books3†
OpenWebText2
ArXiv
Github
FreeLaw
Stack Exchange
USPTO Backgrounds
PubMed Abstracts
Gutenberg (PG-19)†
OpenSubtitles†
Wikipedia (en)†
DM Mathematics†
Ubuntu IRC
BookCorpus2
EuroParl†
HackerNews
YoutubeSubtitles
PhilPapers
NIH ExPorter
Enron Emails†
The Pile
The Pile- An 800GB Dataset of Diverse Text for Language Modeling