6

Investigating and Documenting the
Datasets

As the scale of machine learning research has
grown, scrutiny has been placed on the ever
larger datasets that models are trained on (Prabhu
and Birhane, 2020; Biderman and Scheirer,
2020)

Using fasttext (Suárez et al., 2019a), we deter-
mine that the Pile is 97.4% English. We note that
due to issues with language identiﬁcation, partic-
ularly with rare languages Caswell et al. (2020),
this methodology provides only a rough estimate
for English content and no reliable conclusions for
low-resource languages can be drawn.

While this issue has been raised within AI ethics
and bias research (Hovy and Spruit, 2016; Hutchin-
son et al., 2020; Blodgett et al., 2020), it has not
been a focal point of concern within the language
modeling community. Despite the proliferation
of work exploring and documenting issues with
datasets (Gebru et al., 2018; Bender and Friedman,

10
The Pile- An 800GB Dataset of Diverse Text for Language Modeling