We note that this metric is only a proxy for similar-
ity, and that it could be confounded by dataset spe-
ciﬁc scaling effects. Although our results largely
accord with expectations, there are some puzzling
results, like the datasets on which GPT-3 outper-
formed GPT-2 Pile. We hypothesize that GPT-3
learns to be so good at these datasets that train-
ing on them explicitly does not notably beneﬁt the
model’s performance. We leave a more rigorous
analysis of these effects for future work.

4 Evaluation

To conﬁrm the effectiveness of the Pile for im-
proving language modeling quality, we train
architecturally-identical 1.3 billion parameter mod-
els based on those in Brown et al. (2020) on dif-
ferent datasets and evaluate on the WikiText and
LAMBADA tasks as benchmarks of language mod-
eling ability. We also report results on the Pile
as a measure of more cross-domain generaliza-
tion.

4.1 Methodology
The Pile- An 800GB Dataset of Diverse Text for Language Modeling