1

Introduction

Recent breakthroughs in general-purpose language
modeling have demonstrated the effectiveness of
training massive models on large text corpora for
downstream applications (Radford et al., 2019;
Shoeybi et al., 2019; Raffel et al., 2019; Rosset,
2019; Brown et al., 2020; Lepikhin et al., 2020). As
the ﬁeld continues to scale up language model train-
ing, the demand for high-quality massive text data
will continue to grow (Kaplan et al., 2020).

The growing need for data in language modeling
has caused most existing large-scale language mod-
els to turn to the Common Crawl for most or all of
their data (Brown et al., 2020; Raffel et al., 2019).
While training on the Common Crawl has been
effective, recent work has shown that dataset di-

1https://pile.eleuther.ai/

1
The Pile- An 800GB Dataset of Diverse Text for Language Modeling