2.9 USPTO Backgrounds
USPTO Backgrounds is a dataset of background
sections from patents granted by the United States
Patent and Trademark Ofﬁce, derived from its pub-
lished bulk archives5. A typical patent background
lays out the general context of the invention, gives
an overview of the technical ﬁeld, and sets up the
framing of the problem space. We included USPTO
Backgrounds because it contains a large volume of
technical writing on applied subjects, aimed at a
non-technical audience.

2.10 Wikipedia (English)
Wikipedia is a standard source of high-quality text
for language modeling.
In addition to being a
source of high quality, clean English text, it is also
valuable as it is written in expository prose, and
spans many domains.
The Pile- An 800GB Dataset of Diverse Text for Language Modeling