From that, each publication is downloaded. Some
entries do not exist, or have been removed by
the authors. Papers with text are extracted using
pdfbox, and papers with non-machine readable
text are ignored. Non-English language publica-
tions are kept, and the metadata reﬂects the lan-
guage reported by the OAI-MPH XML. The text is
ﬁltered with pdf_filter.py from PDFextract,
and we discard any papers with less than 1000 char-
acters.24

C.21 NIH Grant abstracts:

ExPORTER

The NIH provides a bulk-data repository for
awarded applications through the ExPORTER ser-
vice covering the ﬁscal years 1985–present. These
data come from the NIH, but also other other Health
and Human Services agencies (ACF, AHRQ, CDC,
HRSA, FDA), and the VA. Additionally, the NIH

22https://github.com/jdepoix/

youtube-transcript-api

23https://github.com/vphill/

pyoaiharvester/

24https://github.com/sdtblck/PDFextract

26
The Pile- An 800GB Dataset of Diverse Text for Language Modeling