Our preferred metric is bits per UTF-8 encoded
byte (BPB). Bits per byte is preferred over bits per
character or perplexity when using Pile as a met-
ric due to its invariance to different tokenization
schemes and the ambiguity of measuring charac-
ters in Unicode. To compute bits per byte from a
given negative log likelihood loss (cid:96), we compute
BPB = (LT /LB) log2(e(cid:96)) = (LT /LB)(cid:96)/ ln(2),
where LT is the length of the dataset in tokens and
LB is the length of the dataset in UTF-8 encoded
bytes. We ﬁnd that LT /LB is 0.29335 GPT-2-
tokens/byte across the Pile; dataset-speciﬁc values
of LT /LB can be found in Table 7.

3.2 Test Perplexity with GPT-2 and

GPT-3

We compute the test perplexity of the constituent
datasets of the Pile using GPT-2 (Radford et al.,
The Pile- An 800GB Dataset of Diverse Text for Language Modeling