4.2 Results
On traditional language modeling benchmarks, the
Pile improves signiﬁcantly on WikiText and shows
negligible changes in LAMBADA. However, mod-
els trained on Pile improve signiﬁcantly over both
Raw CC and CC-100 on all components of the
Pile, as shown in Table 4. This indicates that mod-
els trained on the Pile have greater cross-domain
generalization capabilities without compromising
performance on traditional benchmarks.

The magnitude of improvement over CC-100 per
set is shown in Figure 4. Unsurprisingly, there
is almost no improvement on Pile-CC. However,
the model trained on the Pile performs signiﬁ-
cantly better than either of the other models on
academic datasets such as ArXiv, Pubmed Central,
FreeLaw, and PhilPapers. It also improves signiﬁ-

11The data was obtained from http://data.statmt.

org/cc-100/.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling