We remove any line beginning with :::, which is
used by pandoc to indicate html classes in mark-
down.

C.6 GitHub
We separate the data gathering process into two
steps:

1. Gathering a list of the desired repositories and

their metadata

2. Extracting all text data useful for language

modeling from each repository
The Pile- An 800GB Dataset of Diverse Text for Language Modeling