There are ﬁve datasets included in the Pile that were
not collected and distributed in a ToS compliant
fashion and for which the authors had no ability
to consent to their data being used. Each of these
datasets are widely used, both in the NLP litera-
ture and the world at large. With the exception
of the YouTube Subtitles dataset, each of these
datasets were published by researchers and are
passed around freely on the internet. The YouTube
Subtitles dataset was created by us for this project,
using a very popular unofﬁcial API that is both
widely used and easily obtainable on Pip, Conda,
and GitHub, among other places. Given the pro-
cessing applied and the difﬁculty of identifying par-
ticular ﬁles in the Pile, we feel that our use of these
datasets does not constitute signiﬁcantly increased
harm beyond that which has already been done by
the widespread publication of these datasets.

7

Implications and Broader
Impacts
The Pile- An 800GB Dataset of Diverse Text for Language Modeling