While the majority of documents in the Pile are
short, there is a long tail of very long documents
(Figure 5).

Since the GPT-2 BPE tokenizer is trained on Web-
Text, the mean bytes per token is also a very rough
indicator of how syntactically different each Pile
component is from WebText. For instance, datasets
like NIH ExPorter, OpenWebText2 and Books3

consist largely of ordinary text in a similar distri-
bution to WebText, which is reﬂected in a greater
number of bytes per token. On the other hand,
many of the sets with the lowest bytes per token
are those which consist in large part of non-text
content (Github, ArXiv, Stack Exchange, and DM
Mathematics) or languages other than English (Eu-
roParl).
The Pile- An 800GB Dataset of Diverse Text for Language Modeling