0250500750100012501500203040506070AccuracyTriviaQA02505007501000125015005055606570758085HellaSwag025050075010001250150005101520253035NaturalQuestions0250500750100012501500Billion of tokens40424446485052AccuracySIQA0250500750100012501500Billion of tokens50556065707580WinoGrande0250500750100012501500Billion of tokens65.067.570.072.575.077.580.082.5PIQALLaMA 7BLLaMA 13BLLaMA 33BLLaMA 65BChinchillaGender
Religion
Race/Color
Sexual orientation
Age
Nationality
Disability
Physical appearance
Socioeconomic status
Average

LLaMA GPT3 OPT
65.7
68.6
68.6
78.6
67.8
62.9
76.7
76.2
76.2
69.5

62.6
73.3
64.7
76.2
64.4
61.6
76.7
74.6
73.8
67.2

70.6
79.0
57.0
81.0
70.1
64.2
66.7
77.8
71.5
66.6

Table 12: CrowS-Pairs. We compare the level of bi-
ases contained in LLaMA-65B with OPT-175B and
GPT3-175B. Higher score indicates higher bias.
LLaMA- Open and Efficient Foundation Language Models