small medium large
17.9
26.5
10.3
7.6
18.5
27.9
15.1
23.5
9.7
13.7
16.5
16.7
10.8
15.9
9.0
13.7
16.6
11.5
13.3
20.9
22.8
37.8
16.6
22.1
27.0
17.5
8.3
9.9
22.6
33.3
17.6
25.1
63.9
33.5
30.9
43.7
16.2
25.3
30.1
20.4
15.5
23.2
18.6
22.0
18.4
13.1

20.3
8.3
21.3
17.5
10.7
8.1
11.6
9.4
13.0
15.4
24.9
18.1
19.8
8.7
26.1
20.0
41.9
28.3
18.8
23.0
17.7
15.4
13.3

xl
16.7
7.1
17.0
13.6
9.0
13.5
10.1
8.4
10.6
12.1
24.3
15.8
16.8
7.9
20.9
16.3
27.7
32.4
14.8
18.6
14.2
18.1
12.0

Table 9: Full Test Perplexity of the Pile using GPT-2.

Female
little
married
sexual
happy
young
soft
hot
tiny
older
black
emotional
worried
nice
live
lesbian

Male
general
military
united
political
federal
great
national
guilty
criminal
former
republican
american
major
such
offensive

Atheist
religious
agnostic
such
liberal
likely
much
less
least
political
moral
scientiﬁc
rational
skeptic
skeptical
intellectual

Table 10: Top 15 most biased adjectives/adverbs for each gender
The Pile- An 800GB Dataset of Diverse Text for Language Modeling