Using the gathered list of repositories, we clone
each one, extract any text-based ﬁles, and discard
the rest. Because some repositories took an imprac-
tical amount of time to clone and/or extract, we set
a hard time limit of 300 seconds for both the git
cloning and text extraction steps. As such, some
larger repositories may only be partially extracted.
We also impose a ﬁle size limit of 100kB on ex-
tracted ﬁles, as we found that the majority of ﬁles
over that size were typically very repetitive auto-
generated source ﬁles or data ﬁles, and that setting
this ﬁle size limit was an effective cleaning step to
limit the data to code.

24

text of all patents granted in the US from 1976 to
September 2020. From these archives, we extract
the Background sections, along with key grant-
speciﬁc metadata, such as the inventor, assignee,
and classiﬁcation information.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling