1https://pile.eleuther.ai/

1

versity leads to better downstream generalization
capability (Rosset, 2019). Additionally, large-scale
language models have been shown to effectively
acquire knowledge in a novel domain with only
relatively small amounts of training data from that
domain (Rosset, 2019; Brown et al., 2020; Carlini
et al., 2020). These results suggest that by mix-
ing together a large number of smaller, high qual-
ity, diverse datasets, we can improve the general
cross-domain knowledge and downstream general-
ization capabilities of the model compared to mod-
els trained on only a handful of data sources.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling