10

tives (Seck et al., 2018; Costa-jussà et al., 2020;
Thieme et al., 2020). The second, the data state-
ments methodology (Bender and Friedman, 2018),
was proposed speciﬁcally for natural language pro-
cessing and has been well received by the NLP
community. Our datasheet and data statement will
be featured in the GitHub repository where the
code for the Pile is stored and will also be available
as separate documents on arXiv (Biderman et al.,
2021; Biderman, 2021).

In addition to the datasheet and data statement,
there is additional information that may be helpful
to people training language models that these doc-
uments do not cover. In the rest of this section we
investigate and document in greater detail some of
this additional contextual information.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling