The Word Embedding Association Test shows that GloVe and word2vec word embeddings exhibit human-like implicit biases based on gender, race, and other social constructs (Caliskan et al., 2017). Meanwhile, research on learning reusable text representations has begun to explore sentence-level texts, with some sentence encoders seeing enthusiastic adoption. Accordingly, we extend the Word Embedding Association Test to measure bias in sentence encoders. We then test several sentence encoders, including state-of-the-art methods such as ELMo and BERT, for the social biases studied in prior work and two important biases that are difficult or impossible to test at the word level. We observe mixed results including suspicious patterns of sensitivity that suggest the test's assumptions may not hold in general. We conclude by proposing directions for future work on measuring bias in sentence encoders. Word embeddings quickly achieved wide adoption in natural language processing (NLP), precipitating the development of efficient, word-level neural models of human language. However, prominent word embeddings such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) encode systematic biases against women and black people (Bolukbasi et al., 2016; Garg et al., 2018, i.a.), implicating many NLP systems in scaling up social injustice. We investigate whether sentence encoders, which extend the word embedding approach to sentences, are similarly biased. 1 The previously developed Word Embedding Association Test (WEAT; Caliskan et al., 2017) measures bias in word embeddings by comparing two sets of target-concept words to two sets of attribute words. We propose a simple generaliza-1 While encoder training data may contain perspectives from outside the U.S., we focus on biases in U.S. contexts. tion of WEAT to phrases and sentences: the Sentence Encoder Association Test (SEAT). We apply SEAT to sentences generated by inserting individual words from Caliskan et al.'s tests into simple templates such as This is a[n] < word > . To demonstrate the new potential of a sentence-level approach and advance the discourse on bias in NLP, we also introduce tests of two biases that are less amenable to word-level representation: the angry black woman stereotype (Collins, 2004; Madison, 2009; Harris-Perry, 2011; hooks, 2015; Gillespie, 2016) and a double bind on women in professional settings (Heilman et al., 2004). The use of sentence-level contexts also facilitates testing the impact of different experimental designs. For example, several of Caliskan et al.'s tests rely on given names associated with European American and African American people or rely on terms referring to women and men as groups (such as woman and man). We explore the effect of using given names versus group terms by creating alternate versions of several bias tests that swap the two. This is not generally feasible with WEAT, as categories like African Americans lack common single-word group terms. We find varying evidence of human-like bias in sentence encoders using SEAT. Sentence-to-vector encoders largely exhibit the angry black woman stereotype and Caliskan biases, and to a lesser degree the double bind biases. Recent sentence encoders such as BERT (Devlin et al., 2018) display limited evidence of the tested biases. However, while SEAT can confirm the existence of bias, negative results do not indicate the model is bias-free. Furthermore, discrepancies in the results suggest that the confirmed biases may not generalize beyond the specific words and sentences in our test data, and in particular that cosine similarity may not be a suitable measure of representational similarity in recent models, indicating a need for alternate bias detection techniques. The Word Embedding Association Test WEAT imitates the human implicit association test (Greenwald et al., 1998) for word embeddings, measuring the association between two sets of target concepts and two sets of attributes. Let X and Y be equal-size sets of target concept embeddings and let A and B be sets of attribute embeddings. The test statistic is a difference between sums over the respective target concepts, s ( X, Y, A, B ) = (cid:2)(cid:80) x  X s ( x, A, B )  (cid:80) y  Y s ( y, A, B ) (cid:3) , where each addend is the difference between mean cosine similarities of the respective attributes, s ( w, A, B ) = (cid:2) mean a  A cos( w, a )  mean b  B cos( w, b ) (cid:3) A permutation test on s ( X, Y, A, B ) is used to compute the significance of the association between ( A, B ) and ( X, Y ) , p = Pr [ s ( X i , Y i , A, B ) > s ( X, Y, A, B )] , where the probability is computed over the space of partitions ( X i , Y i ) of X  Y such that X i and Y i are of equal size, and a normalized difference of means of s ( w, A, B ) is used to measure the magnitude of the association (the effect size; Caliskan et al., 2017), d = mean x  X s ( x, A, B )  mean y  Y s ( y, A, B ) std dev w  X  Y s ( w, A, B ) . Controlling for significance, a larger effect size re-flects a more severe bias. We detail our implementations in the supplement. The Sentence Encoder Association Test SEAT compares sets of sentences, rather than sets of words, by applying WEAT to the vector representation of a sentence. Because SEAT operates on fixed-sized vectors and some encoders produce variable-length vector sequences, we use pooling as needed to aggregate outputs into a fixed-sized vector. We can view WEAT as a special case of SEAT in which the sentence is a single word. In fact, the original WEAT tests have been run on the Universal Sentence Encoder (Cer et al., 2018). To extend a word-level test to sentence contexts, we slot each word into each of several semantically bleached sentence templates such as This is < word > .,  < word > is here., This will < word > ., and  < word > are things.. These templates make heavy use of deixis and are designed to convey little specific meaning beyond that of the terms inserted into them. 2 For example, the word version of Caliskan Test 3 is illustrated in Table 1 and the sentence version is illustrated in Table 2. We choose this design to focus on the associations a sentence encoder makes with a given term rather than those it happens to make with the contexts of that term that are prevalent in the training data; a similar design was used in a recent sentiment analysis evaluation corpus stratified by race and gender (Kiritchenko and Mohammad, 2018). To facilitate future work, we publicly release code for SEAT and all of our experiments. 3 3 Biases Tested Caliskan Tests We first test whether the sentence encoders reproduce the same biases that word embedding models exhibited in Caliskan et al. (2017). These biases correspond to past social psychology studies of implicit associations in human subjects. 4 We apply both the original 2 See the supplement for further details and examples. 3 http://github.com/W4ngatang/sent-bias 4 See Greenwald et al. (2009) for a review of this work. Angry Black Woman Stereotype In the Sapphire or angry black woman (ABW) stereotype, black women are portrayed as loud, angry, and imposing (Collins, 2004; Madison, 2009; Harris-Perry, 2011; hooks, 2015; Gillespie, 2016). This stereotype contradicts common associations made with the ostensibly race-neutral (unmarked) category of women (Bem, 1974), suggesting that that category is implicitly white. Intersectionality reveals that experiences considered common to women are not necessarily shared by black women, who are marginalized both among women and among black people (Crenshaw, 1989). Recently, intersectionality has been demonstrated in English Wikipedia using distributional semantic word representations (Herbelot et al., 2012), and in the disparate error rates of machine learning technologies like face recognition (Buolamwini and Gebru, 2018). To measure sentence encoders' reproduction of the angry black woman stereotype, we create a test whose target concepts are black-identifying and white-identifying female given names from Sweeney (2013, Table 1) and whose attributes are adjectives used in the discussion of the stereotype in Collins (2004, pp. 87-90) and their antonyms. We also produce a version of the test with attributes consisting of terms describing black women and white women as groups, as well as sentence versions in which attribute and target concept terms are inserted in sentence templates. Double Binds Women face many double binds , contradictory or unsatisfiable expectations of femininity and masculinity (Stone and Lovejoy, 2004; Harris-Perry, 2011; Mitchell, 2012). If women clearly succeed in a male gender-typed job, they are perceived less likable and more hostile than men in similar positions; if success is ambiguous, they are perceived less competent and achievement-oriented than men. Both outcomes can interfere in performance evaluations (Heilman et al., 2004), contributing to the glass ceiling impeding women's career advancement. 5 We test this double bind in sentence encoders by translating Heilman et get concepts by names of women and men, respectively, in the single sentence template  < word > is an engineer with superior technical skills.; the attributes are likable and non-hostile terms, based on Heilman et al.'s design, in the sentence template The engineer is < word > . In the second, we use the shortened target concept sentence template  < word > is an engineer and fill the attribute templates from before with competent and achievement-oriented terms based on Heilman et al.'s design. 6 We refer to these tests as semantically unbleached because the context contains important information about the bias. We produce two variations of these tests: word-level tests in which target concepts are names in isolation and attributes are adjectives in isolation, as well as corresponding semantically bleached sentence-level tests. These control conditions allow us to probe the extent to which observed associations are attributable to gender independent of context. We apply SEAT to seven sentence encoders (listed in Table 3) including simple bag-of-words encoders, sentence-to-vector models, and state-of-the-art sequence models. 7 For all models, we use publicly available pretrained parameters. Table 4 shows effect size and significance at 0.01 before and after applying the Holm-Bonferroni multiple testing correction (Holm, 1979) for a subset of tests and models; complete results are provided in the supplement. 8 6 We consider other formulations in the supplement. 7 We provide further details and explore variations on these model configurations in the supplement. 8 We use the full set of tests and models when comput-Test Context CBoW InferSent GenSen USE ELMo GPT BERT C1: Flowers/Insects word 1 . Specifically, we select Caliskan Test 1 associating flowers/insects with pleasant/unpleasant, Test 3 associating European/African American names with pleasant/unpleasant, and Test 6 associating male/female names with career/family, as well as the angry black woman stereotype and the competent and likable double bind tests. We observe that tests based on given names more often find a significant association than those based on group terms; we only show the given-name results here. We find varying evidence of bias in sentence encoders according to these tests. Bleached sentence-level tests tend to elicit more significant associations than word-level tests, while the latter tend to have larger effect sizes. We find stronger evidence for the Caliskan and ABW stereotype tests than for the double bind. After the multiple testing correction, we only find evidence of the double bind in bleached, sentence-level competent control tests; that is, we find women are associated with incompetence independent of context. 9 Some patterns in the results cast doubt on the reasonableness of SEAT as an evaluation. For instance, Caliskan Test 7 (association between math/art and male/female ) and Test 8 ( sci-ence/art and male/female ) elicit counterintuitive results from several models. These tests have the same sizes of target concept and attribute sets. For CBoW on the word versions of those tests, we see p -values of 0.016 and 10  2 , respectively. ing the multiple testing correction, including those only presented in the supplement. 9 However, the double bind results differ across models; we show no significant associations for ELMo or GPT and only one each for USE and BERT. On the sentence versions, we see p -values of 10  5 for both tests. Observing similar p -values agrees with intuition: The math/art association should be similar to the science/art association because they instantiate a disciplinary dichotomy between math/science and arts/language (Nosek et al., 2002). However, for BERT on the sentence version, we see discrepant p -values of 10  5 and 0.14; for GenSen, 0.12 and 10  3 ; and for GPT, 0.89 and 10  4 . Caliskan Tests 3, 4, and 5 elicit even more counterintuitive results from ELMo. These tests measure the association between European Amer-ican/African American and pleasant/unpleasant . Test 3 has larger attribute sets than Test 4, which has larger target concept sets than Test 5. Intuitively, we expect increasing p -values across Tests 3, 4, and 5, as well-designed target concepts and attributes of larger sizes should yield higher-power tests. Indeed, for CBoW, we find increasing p values of 10  5 , 10  5 , and 10  4 on the word versions of the tests and 10  5 , 10  5 , and 10  2 on the sentence versions, respectively. 10 However, for ELMo, we find decreasing p -values of 0 . 95 , 0 . 45 , and 0 . 08 on the word versions of the tests and 1 , 0 . 97 , and 10  4 on the sentence versions. We interpret these results as ELMo producing substantially different representations for conceptually similar words. Thus, SEAT's assumption that the sentence representations of each target concept and attribute instantiate a coherent concept appears invalid. At face value, our results suggest recent sentence encoders exhibit less bias than previous models do, at least when bias is considered from a U.S. perspective and measured using the specific tests we have designed. However, we strongly caution against interpreting the number of significant associations or the average significant effect size as an absolute measure of bias. Like WEAT, SEAT only has positive predictive ability: It can detect presence of bias, but not its absence. Considering that these representations are trained without explicit bias control mechanisms on naturally occurring text, we argue against interpreting a lack of evidence of bias as a lack of bias. Moreover, the counterintuitive sensitivity of SEAT on some models and biases suggests that biases revealed by SEAT may not generalize beyond the specific words and sentences in our test data. That is, our results invalidate the assumption that each set of words or sentences in our tests represents a coherent concept/attribute (like African American or pleasant ) to the sentence encoders; hence, we do not assume the encoders will exhibit similar behavior on other potential elements of those concepts/attributes (other words or sentences representing, for example, African American or pleasant ). One possible explanation of the observed sensitivity at the sentence level is that, from the sentence encoders' view, our sentence templates are not as semantically bleached as we expect; small variations in their relative frequencies and interactions with the terms inserted into them may be undermining the coherence of the concepts/attributes they implement. Another possible explanation that also accounts for the sensitivity observed in the word-level tests is that cosine similarity is an inadequate measure of text similarity for sentence encoders. If this is the case, the biases revealed by SEAT may not translate to biases in downstream applications. Future work could measure bias at the application level instead, following Bailey and Deery (2018)'s recommendation based on the tension between descriptive and normative correctness in representations. The angry black woman stereotype represents an intersectional bias, a phenomenon not well anticipated by an additive model of racism and sexism (Crenshaw, 1989). Previous work has modeled biases at the intersection of race and gender in distributional semantic word representations (Her-belot et al., 2012), natural language inference data (Rudinger et al., 2017), and facial recognition systems (Buolamwini and Gebru, 2018), as well as at the intersection of dialect and gender in automatic speech recognition (Tatman, 2017). We advocate for further consideration of intersectionality in future work in order to avoid reproducing the erasure of multiple minorities who are most vulnerable to bias. We have developed a simple sentence-level extension of an established word embedding bias instrument and used it to measure the degree to which pretrained sentence encoders capture a range of social biases, observing a large number of significant effects as well as idiosyncrasies suggesting limited external validity. This study is preliminary and leaves open to investigation several design choices that may impact the results; future work may consider revisiting choices like the use of semantically bleached sentence inputs, the aggregation applied to models that represent sentences with sequences of hidden states, and the use of cosine similarity between sentence representations. We challenge researchers of fairness and ethics in NLP to critically (re-)examine their methods; looking forward, we hope for a deeper consideration of the social contexts in which NLP systems are applied. We are grateful to Carolyn Rose, Jason Phang, Sebastien Jean, Thibault Fevry, Katharina Kann, and Benjamin Van Durme for helpful conversations concerning this work and to our reviewers for their thoughtful feedback. CM is funded by IARPA MATERIAL; RR is funded by DARPA AIDA; AW is funded by an NSF fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of IARPA, DARPA, NSF, or the U.S. Government.
abstain abstain objective method result objective abstain abstain objective abstain objective abstain method abstain other abstain abstain abstain objective abstain result abstain abstain abstain result abstain abstain other abstain method abstain method method abstain method abstain abstain method method method abstain method other abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result result abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain result result abstain method result result method abstain abstain abstain abstain abstain abstain objective result method other other other other