experiences of members of communities affected
by these systems. Second, we recommend that
the power relations between technologists and
such communities be interrogated and reimagined.
Researchers have pointed out that algorithmic
fairness techniques, by proposing incremental
technical mitigations—e.g., collecting new datasets
or training better models—maintain these power
relations by (a) assuming that automated systems
should continue to exist, rather than asking
whether they should be built at all, and (b) keeping
development and deployment decisions in the
hands of technologists (Bennett and Keyes, 2019;
Cifor et al., 2019; Green, 2019; Katell et al., 2020).
There are many disciplines for researchers and
practitioners to draw on when pursuing these
directions. For example, in human–computer
interaction, Hamidi et al. (2018) study transgender
people’s experiences with automated gender
recognition systems in order to uncover how
these systems reproduce structures of transgender
exclusion by redeﬁning what it means to perform
gender “normally.” Value-sensitive design provides
a framework for accounting for the values of different stakeholders in the design of technology (e.g.,
Friedman et al., 2006; Friedman and Hendry, 2019;
Le Dantec et al., 2009; Yoo et al., 2019), while
We survey 146 papers analyzing "bias" in NLP systems, finding that their
motivations are often vague, inconsistent, and lacking in normative reasoning,
despite the fact that analyzing "bias" is an inherently normative process. We
further find that these papers' proposed quantitative techniques for measuring
or mitigating "bias" are poorly matched to their motivations and do not engage
with the relevant literature outside of NLP. Based on these findings, we
describe the beginnings of a path forward by proposing three recommendations
that should guide work analyzing "bias" in NLP systems. These recommendations
rest on a greater recognition of the relationships between language and social
hierarchies, encouraging researchers and practitioners to articulate their
conceptualizations of "bias"---i.e., what kinds of system behaviors are
harmful, in what ways, to whom, and why, as well as the normative reasoning
underlying these statements---and to center work around the lived experiences
of members of communities affected by NLP systems, while interrogating and
reimagining the power relations between technologists and such communities.
Su Lin Blodgett Solon Barocas Hal Daumé III Hanna Wallach