3. We use YoutubeTranscriptApi22 to
gather all human generated closed captions
for every available language for each video.
To align each language in parallel, we split
the captions for each language into parallel
minute-long sections by timestamp, and ar-
range each language in a random order within
these sections, appending the language as a
header to each minute-long section to provide
context. If only a single language is available,
the output is just the subtitles, with no header
appended.

In total, subtitles for 173,651 videos were gath-
ered.

C.20 PhilPapers
The PhilPapers (PP) are indexed using OAI-MPH,
the Open Archives Initiative Protocol for Metadata
Harvesting. As such, the ﬁrst step to collect the
data is to get the XML for all links. This was done
using pyoaiharvester.23
The Pile- An 800GB Dataset of Diverse Text for Language Modeling