5.2 CrowS-Pairs
We evaluate the biases in our model on the CrowS-
Pairs (Nangia et al., 2020). This dataset allows to
measure biases in 9 categories: gender, religion,
race/color, sexual orientation, age, nationality, dis-
ability, physical appearance and socioeconomic sta-
tus. Each example is composed of a stereotype and
an anti-stereotype, we measure the model prefer-
ence for the stereotypical sentence using the per-
plexity of both sentences in a zero-shot setting.
Higher scores thus indicate higher bias. We com-
pare with GPT-3 and OPT-175B in Table 12.

LLaMA compares slightly favorably to both
models on average. Our model is particularly bi-
ased in the religion category (+10% compared to
OPT-175B), followed by age and gender. We ex-
pect these biases to come from CommonCrawl de-
spite multiple ﬁltering steps.
LLaMA- Open and Efficient Foundation Language Models