C.11 Project Gutenberg
No additional details.

C.12 OpenSubtitles
To create the text dataset, we simply extract the
subtitle text from each XML ﬁle in the English
language dataset provided by Tiedemann (2016),
discarding any provided metadata.

C.13 Wikipedia (English)
We use the wikipedia/20200301.en dataset
from TensorFlow Datasets.20 We prepend the ti-
tle to the body of each article, separated by two
newlines.

C.14 DeepMind Mathematics
We include instances from the Easy, Medium,
and Hard components of DeepMind Mathemat-
ics, breaking each curriculum item (such as
algebra__polynomial_roots) into 8 KiB
chunks.

20https://www.tensorflow.org/datasets/

catalog/wikipedia#wikipedia20200301en

C.15 Ubuntu IRC
We processed all logs from July 5, 2004 through
September 1, 2020.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling