the relationships between language and social
hierarchies; without this grounding, researchers
and practitioners risk measuring or mitigating
only what is convenient to measure or mitigate,
rather than what is most normatively concerning.
More speciﬁcally, we recommend that work
analyzing “bias” in NLP systems be reoriented
around the following question: How are social
hierarchies, language ideologies, and NLP systems
coproduced? This question mirrors Benjamin’s
(2020) call to examine how “race and technology
are coproduced”—i.e., how racial hierarchies, and
the ideologies and discourses that maintain them,
create and are re-created by technology. We recommend that researchers and practitioners similarly
ask how existing social hierarchies and language
ideologies drive the development and deployment
of NLP systems, and how these systems therefore
reproduce these hierarchies and ideologies. As
a starting point for reorienting work analyzing
“bias” in NLP systems around this question, we
3Language ideologies encompass much more than this; see,
e.g., Lippi-Green (2012), Alim et al. (2016), Rosa and Flores
(2017), Rosa and Burdick (2017), and Charity Hudley (2017).provide the following concrete research questions:
We survey 146 papers analyzing "bias" in NLP systems, finding that their
motivations are often vague, inconsistent, and lacking in normative reasoning,
despite the fact that analyzing "bias" is an inherently normative process. We
further find that these papers' proposed quantitative techniques for measuring
or mitigating "bias" are poorly matched to their motivations and do not engage
with the relevant literature outside of NLP. Based on these findings, we
describe the beginnings of a path forward by proposing three recommendations
that should guide work analyzing "bias" in NLP systems. These recommendations
rest on a greater recognition of the relationships between language and social
hierarchies, encouraging researchers and practitioners to articulate their
conceptualizations of "bias"---i.e., what kinds of system behaviors are
harmful, in what ways, to whom, and why, as well as the normative reasoning
underlying these statements---and to center work around the lived experiences
of members of communities affected by NLP systems, while interrogating and
reimagining the power relations between technologists and such communities.
Su Lin Blodgett Solon Barocas Hal Daumé III Hanna Wallach