papers analyzing “bias” in NLP systems developed
for the same task often conceptualize it differently.
For example, the following system behaviors
are all understood to be self-evident statements of
“racial bias”: (a) embedding spaces in which embeddings for names associated with African Americans
are closer (compared to names associated with
European Americans) to unpleasant words than
pleasant words (Caliskan et al., 2017); (b) sentiment analysis systems yielding different intensity
scores for sentences containing names associated
with African Americans and sentences containing
names associated with European Americans (Kiritchenko and Mohammad, 2018); and (c) toxicityarXiv:2005.14050v2  [cs.CL]  29 May 2020
detection systems scoring tweets containing features associated with African-American English as
more offensive than tweets without these features
(Davidson et al., 2019; Sap et al., 2019). Moreover,
some of these papers focus on “racial bias”
expressed in written text, while others focus on
“racial bias” against authors. This use of imprecise
terminology obscures these important differences.
We survey 146 papers analyzing “bias” in NLP
systems, ﬁnding that their motivations are often
vague and inconsistent. Many lack any normative
We survey 146 papers analyzing "bias" in NLP systems, finding that their
motivations are often vague, inconsistent, and lacking in normative reasoning,
despite the fact that analyzing "bias" is an inherently normative process. We
further find that these papers' proposed quantitative techniques for measuring
or mitigating "bias" are poorly matched to their motivations and do not engage
with the relevant literature outside of NLP. Based on these findings, we
describe the beginnings of a path forward by proposing three recommendations
that should guide work analyzing "bias" in NLP systems. These recommendations
rest on a greater recognition of the relationships between language and social
hierarchies, encouraging researchers and practitioners to articulate their
conceptualizations of "bias"---i.e., what kinds of system behaviors are
harmful, in what ways, to whom, and why, as well as the normative reasoning
underlying these statements---and to center work around the lived experiences
of members of communities affected by NLP systems, while interrogating and
reimagining the power relations between technologists and such communities.
Su Lin Blodgett Solon Barocas Hal Daumé III Hanna Wallach