!

!

The most frequently occurring 13-grams were
character repetitions used for styling such as
“-- --”, “* * * *”, “!
!”, at 11
million, 5.8 million and 1.1 million respectively.
Other characters used in this manner include the
following: “# .
In the 264k count
range, we see repetitions of badly formatted
HTML escape characters “; &nbsp”, “; amp”.
Boilerplate from standard forum software appears
around the 180k occurrences range, such as the
following: “select the forum that you
want to visit from the selection
below”.

> ?”.

Overall, a large amount of common HTML and
CSS is included in the top 1000, along with boil-
erplate text from Amazon Afﬁliate Advertising,

Component

Pile-CC
PubMed Central
Books3
OpenWebText2
Arxiv
Github
FreeLaw
StackExchange
USPTO Backgrounds
PubMed Abstracts
Gutenberg (PG-19)
OpenSubtitles
Wikipedia (en)
DM Mathematics
Ubuntu IRC
BookCorpus2
EuroParl
HackerNews
YoutubeSubtitles
PhilPapers
NIH ExPorter
Enron Emails
The Pile- An 800GB Dataset of Diverse Text for Language Modeling