18https://arxiv.org/help/bulk_data_s3

(a) OpenWebText2

(b) Full Pile

Figure 9: Score distribution of documents from Common Crawl given different classiﬁer training data.

Because we wanted to limit the size of the overall
Pile, we randomly sampled 95.0 GiB of the 630.64
GiB of Github data we collected in total and leave
quality ﬁltering to future work.

However, we believe code generation will be an in-
creasingly important component of language mod-
els as they continue to scale up and increase in their
ability to generalize. As such, we hope to extend
this dataset in future work.

C.7 FreeLaw
We download the court opinions data in bulk from
CourtListener,19 and extract the raw text using
BeautifulSoup.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling