28

plexity score. The perplexity for the whole Pile
is computed by aggregating over the constituent
datasets (i.e. weighted by dataset size, not a simple
average of dataset perplexities). Both GPT-2 and
GPT-3 share the same tokenizer and vocabulary,
making the perplexity scores directly comparable.
We use the Hugging Face (Wolf et al., 2020) im-
plementation of GPT-2, and the OpenAI API for
GPT-3. The davinci model in the OpenAI API
is presumed to correspond to a 175B parameter
version of GPT-3.

In Table 8 we show the test set perplexities (i.e. not
normalized by UTF-8 length, as in Table 2). Be-
cause of the costs associated with using the OpenAI
API, we compute test perplexities on only one-tenth
of the test set in Tables 8 and Table 2. Speciﬁcally,
we randomly sample one-tenth of the documents of
each dataset except for three: Ubuntu IRC, Book-
Corpus2, and PhilPapers.
In Table 9, we show
test perplexity computed on the full test set on all
GPT-2 models.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling