23https://github.com/vphill/

pyoaiharvester/

24https://github.com/sdtblck/PDFextract

26

provides a legacy data format named CRISP for
awarded applications during the ﬁscal years 1970–
2009.

We merged both the ExPORTER and CRISP data
to form a consolidated dataset of awarded appli-
cations. Entries were deduplicated based off their
application ID, and excluded if their abstract text
was missing or too short. Small grants, especially
administrative ones, consisted solely of short boil-
erplate. For this reason, we further deduplicated on
abstract text. All grants types were considered, in-
cluding new applications (Application Type Code
1) and renewals (Application Type Code 2) as the
text differed enough to provide novel input. The
text was then minimally parsed to remove admin-
istrative boilerplate, (ex. most old awards contain
some variation of “description: (provided by appli-
cant)"). In total, there were 939,668 grant applica-
tion abstracts added.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling