Online textsacross genres, registers, domains, and stylesare riddled with human stereotypes, expressed in overt or subtle ways. Word embeddings, trained on these texts, perpetuate and amplify these stereotypes, and propagate biases to machine learning models that use word embeddings as features. In this work, we propose a method to debias word embeddings in multiclass settings such as race and religion, extending the work of (Boluk-basi et al., 2016) from the binary setting, such as binary gender. Next, we propose a novel methodology for the evaluation of multiclass debiasing. We demonstrate that our multiclass debiasing is robust and maintains the efficacy in standard NLP tasks. In addition to possessing informative features useful for a variety of NLP tasks, word embeddings reflect and propagate social biases present in training corpora (Caliskan et al., 2017; Garg et al., 2018). Machine learning systems that use embeddings can further amplify biases (Barocas and Selbst, 2016; Zhao et al., 2017), discriminating against users, particularly those from disadvantaged social groups. (Bolukbasi et al., 2016) introduced a method to debias embeddings by removing components that lie in stereotype-related embedding subspaces. They demonstrate the effectiveness of the approach by removing gender bias from word2vec embeddings (Mikolov et al., 2013), preserving the utility of embeddings and potentially alleviating biases in downstream tasks. However, this method was only for binary labels (e.g., male/female), whereas most real-world demographic attributes, * Equal contributions  Work done while at CMU and The Microsoft AI Development Acceleration Program Gender Biased Analogies man  doctor woman  nurse woman  receptionist man  supervisor woman  secretary man  principal Racially Biased Analogies black  criminal caucasian  police asian  doctor caucasian  dad caucasian  leader black  led Religiously Biased Analogies muslim  terrorist christian  civilians jewish  philanthropist christian  stooge christian  unemployed jewish  pensioners Table 1: Examples of gender, racial, and religious biases in analogies generated from word embeddings trained on the Reddit data from users from the USA. including gender, race, religion, are not binary but continuous or categorical, with more than two categories. In this work, we show a generalization of Bolukbasi et al.'s (2016) which enables multiclass debiasing, while preserving utility of embeddings (  3). We train word2vec embeddings using the Reddit L2 corpus (Rabinovich et al., 2018) and apply multiclass debiasing using lexicons from studies on bias in NLP and social science (  4.2). We introduce a novel metric for evaluation of bias in collections of word embeddings (  5). Finally, we validate that the utility of debiased embeddings in the tasks of part-of-speech (POS) tagging, named entity recognition (NER), and POS chunking is on par with off-the-shelf embeddings. As defined by (Bolukbasi et al., 2016), debiasing word embeddings in a binary setting requires identifying the bias subspace of the embeddings. Components lying in that subspace are then removed from each embedding. (Bolukbasi et al., 2016) define the gender subspace using defining sets of words, where the words in each set represent different ends of the bias. For example, in the case of gender, one defining set might be the gendered pronouns { he , she } and another set might be the gendered nouns { man , woman } . The gender subspace is then computed from these defining sets by 1) computing the vector differences of the word embeddings of words in each set from the set's mean, and 2) taking the most significant components of these vectors. Following the identification of the gender subspace, one can apply hard or soft debiasing (Bolukbasi et al., 2016) to completely or partially remove the subspace components from the embeddings. Hard debiasing (also called Neutralize and Equalize) involves two steps. First, bias components are removed from words that are not gendered and should not contain gender bias (e.g., doctor , nurse ), and second, gendered word embeddings are centered and their bias components are equalized. For example, in the binary case, man and woman should have bias components in opposite directions, but of the same magnitude. Intuitively, this then ensures that any neutral words are equidistant to any biased words with respect to the bias subspace. More formally, to neutralize , given a bias subspace B spanned by the vectors { b 1 , b 2 , ..., b k } , we compute the component of each embedding in this subspace: w B = k (cid:88) i =1 (cid:104) w , b i (cid:105) b i (1) We then remove this component from words that should be bias-neutral and normalize to get the debiased embedding: w (cid:48) = w  w B (cid:107) w  w B (cid:107) (2) To equalize the embeddings of words in an equality set E , let  = 1 | E | (cid:80) w  E w be the mean embedding of the words in the set and  B be its component in the bias subspace as calculated in Equation 1. Then, for w  E , w (cid:48) = (    B ) + (cid:112) 1  (cid:107)    B (cid:107) 2 w B   B (cid:107) w B   B (cid:107) (3) Note that in both Equations 2 and 3, the new embedding has unit length. Soft debiasing involves learning a projection of the embedding matrix that preserves the inner product between biased and debiased embeddings while minimizing the projection onto the bias subspace of embeddings that should be neutral. Given embeddings W and N which are embeddings for the whole vocabulary and the subset of bias-neutral words respectively, and the bias subspace B obtained in Section 2.1, soft debiasing seeks for a linear transformation A that minimizes the following objective: (cid:107) ( AW ) (cid:124) ( AW )  W (cid:124) W (cid:107) 2 F +  (cid:107) ( AN ) (cid:124) ( A B ) (cid:107) 2 F (4) Minimizing the first term preserves the inner product after the linear transformation A , and minimizing the second term minimizes the projection onto the bias subspace B of embeddings. R is a tunable parameter that balances the two objectives. We now discuss our proposed extension of word embedding debiasing to the multiclass setting. As in the binary setting, debiasing consists of two steps: identifying the bias subspace and removing this component from the set of embeddings. The core contribution of our work is in identifying the bias subspace in a multiclass setting; if we can identify the bias subspace then prior work can be used for multiclass debiasing. Past work has shown that it is possible to linearly separate multiple social classes based on components of word embeddings (Garg et al., 2018). Based on this we hypothesize that there exists some component of these embeddings which can capture multiclass bias. While a multiclass Gender Debiasing MAC P-Value Biased 0.623 N/A Hard Debiased 1.000 1.582e-14 Soft Debiased (  = 0.2) 0.747 1.711e-12 Race Debiasing MAC P-Value Biased 0.892 N/A Hard Debiased 1.009 7.235e-04 Soft Debiased (  = 0.2) 0.985 6.217e-05 Religion Debiasing MAC P-Value Biased 0.859 N/A Hard Debiased 1.004 3.006e-07 Soft Debiased (  = 0.2) 0.894 0.007 Table 2: The associated mean average cosine similarity (MAC) (defined in Section 3.2) and P-Values for debiasing methods for gender, race, and religious bias. problem is inherently not a linearly separable problem, a one versus rest classifier is. Following from this, the computation of a multiclass bias subspace does not have any linear constraints, though it does come with a loss of resolution. As a result we can compute the principal components required to compute the bias subspace by simply adding an additional term for each additional bias class to each defining set. Formally, given defining sets of word embeddings D 1 , D 2 , ..., D n , let the mean of the defining set i be  i = 1 | D i | (cid:80) w  D i w , where w is the word embedding of w . Then the bias subspace B is given by the first k components of the following principal component analysis (PCA) evaluation: PCA   n (cid:91) i =1 (cid:91) w  D i w   i   (5) The number of components k can be empirically determined by inspecting the eigenvalues of the PCA, or using a threshold. Also, note that the defining sets do not have to be the same size. We discuss the robustness of this method later. Following the identification of the bias subspace, we apply the hard Neutralize and Equalize debiasing and soft debiasing method presented in (Bolukbasi et al., 2016) and discussed in Section 2.2 to completely or partially remove the subspace components from the embeddings. For equalization, we take the defining sets to be the equality sets as well. We propose a new metric for the evaluation of bias in collections of words which is simply the mean average cosine similarity (MAC). This approach is motivated by the WEAT evaluation method proposed by (Caliskan et al., 2017) but modified for a multiclass setting. To compute this metric the following data is required: a set of target word embeddings T containing terms that inherently contain some form of social bias (e.g. { church , synagogue , mosque } ), and a set A which contains sets of attributes A 1 , A 2 , ..., AN containing word embeddings that should not be associated with any word embeddings contained in the set T (e.g. { violent , liberal , conservative } ). S ( t , A j ) = 1 N (cid:88) a  A j cos ( t , a ) , (6) where the cosine distance is: cos ( u , v ) = 1  u  v (cid:107) u (cid:107) 2  (cid:107) v (cid:107) 2 . (7) Finally, we define MAC as: MAC ( T, A ) = 1 | T || A | (cid:88) T i  T (cid:88) A j  AS ( T i , A j ) (8) We also perform a paired t -test on the distribution of average cosines used to calculate the MAC. Thus we can quantify the effect of debiasing on word embeddings in T and sets in A . To measure the utility of the debiased word embeddings, we use the tasks of NER, POS tagging, and POS chunking. This is to ensure that the debiasing procedure has not destroyed the utility of the word embeddings. We evaluate test sentences that contain at least one word affected by debiasing. Additionally, we measure the change in performance after replacing the biased embedding matrix by a debiased one, and retraining the model on debiased embeddings. In this section we discuss the different data sources we used for our initial word embeddings, the social bias data used for evaluating bias, and the linguistic data used for evaluating the debiasing process. We used the L2-Reddit corpus (Rabinovich et al., 2018), a collection of Reddit posts and comments by both native and non-native English speakers. The native countries of post authors are determined based on their posts in countryand region-specific subreddits (such as r/Europe and r/UnitedKingdom), and other metadata such as user flairs, which serve as self-identification of the user's country of origin. In this work, we exclusively explore data collected from the United States. This was done to leverage extensive studies of social bias done in the United States. To obtain the initial biased word embeddings, we trained word2vec embeddings (Mikolov et al., 2013) using approximately 56 million sentences. We used the following vocabularies and studies to 1 For gender, we used vocabularies created by (Bolukbasi et al., 2016) and (Caliskan et al., 2017). compile lexicons for bias detection and removal. For race we consulted a number of different sources for each race: Caucasians (Chung-Herrera and Lankau, 2005; Goad, 1998); African Americans (Punyanunt-Carter, 2008; Brown Givens and Monahan, 2005; Chung-Herrera and Lankau, 1 The lexicon used can be found here: https: //docs.google.com/spreadsheets/d/1BQBFLUvB9bnuifxikjrcJNqLA0dx9ZeAWk4kdu_OCUM . 2005; Hakanen, 1995; Welch, 2007; Kawai, 2005); and Asian Americans (Leong and Hayes, 1990; Lin et al., 2005; Chung-Herrera and Lankau, 2005; Osajima, 2005; Garg et al., 2018). Finally, for religion we used the following sources and labels: Christians (Rios et al., 2015; Zuckerman, 2009; Unnever et al., 2005); Jews (Dundes, 1971; Fetzer, 2000); and Muslims (Shry-ock, 2010; Alsultany, 2012; Shaheen, 1997). We evaluate biased and debiased word embeddings on several downstream tasks. Specifically, the CoNLL 2003 shared task (Tjong Kim Sang and De Meulder, 2003) which provides evaluation data for NER, POS tagging, and POS chunking. In this section we review the results of our experiments and discuss what those results mean in the context of this work. We can observe bias in word embeddings in many different ways. However, for the purposes of demonstrating that bias exists in these word embeddings we use the analogy task that was used to demonstrate bias in (Bolukbasi et al., 2016). We observe that bias is present in generated analogies by viewing them directly. A small subset of these analogies are in Table 1 to highlight our findings. We perform our debiasing in the same manner as described in Section 3.1 and calculate the MAC scores and p -values to measure the effects of debiasing. Results are presented in Table 2. Does multiclass debiasing decrease bias? We see that this debiasing procedure categorically moves MAC scores closer to 1.0. This indicates an increase in cosine distance. Further, the associated P-values indicate these changes are statistically significant. This demonstrates that our approach for multiclass debiasing decreases bias. The effects of debiasing on downstream tasks are shown in Table 3. Debiasing can either help or harm performance. For POS tagging there is almost always a decrease in performance. However, for NER and POS chunking, there is a consistent increase. We conclude that these models have learned to depend on some bias subspaces differently. Note that many performance changes are of questionable statistical significance. Does multiclass debiasing preserve semantic utility? We argue the minor changes in Table 3 support the preservation of semantic utility in the multiclass setting, especially compared to gender debiasing which is known to preserve utility (Bolukbasi et al., 2016). Is the calculated bias subspace robust? The bias subspace is at least robust enough to support the above debiasing operations. This is shown by statistically significant changes in MAC scores. Calculating multiclass bias subspace using our proposed approach has drawbacks. For example, in the binary gender case, the extremes of bias subspace reflect extreme male and female terms. However, this is not possible when projecting multiple classes into a linear space. Thus, while we can calculate the magnitude of the bias components, we cannot measure extremes of each class. Additionally, the methods presented here rely on words that represent biases (defining sets) and words that should or should not contain biases (equality sets). These lists are based on data collected specifically from the US. Thus, they may not translate to other countries or cultures. Further, some of these vocabulary terms, while peer reviewed, may be subjective and may not fully capture the bias subspace. Recent work by Gonen and Goldberg (2019) suggests that debiasing methods based on bias component removal are insufficient to completely remove bias in the embeddings, since embeddings with similar biases are still clustered together after bias component removal. Following Gonen and Goldberg's (2019) procedure, we plot the number of neighbors of a particular bias class as a function of the original bias, before and after debiasing in Figure 1 and 2 in the Appendix. In line with Gonen and Goldberg's (2019) findings, simply removing the bias component is insufficient to remove multiclass cluster bias. However, increasing the size of the bias subspace reduces the correlation of the two variables (Table 4 in the Ap-pendix). We showed that word embeddings trained on www.reddit.com data contain multiclass biases. We presented a novel metric for evaluating debiasing procedures for word embeddings. We robustly removed multiclass bias using a generalization of existing techniques. Finally, we showed that this multiclass generalization preserves the utility of embeddings for different NLP tasks. This research was supported by Grant No. IIS1812327 from the United States National Science Foundation (NSF). We also acknowledge several people who contributed to this work: Benjamin Pall for his valuable early support of this work; Elise Romberger who helped edit this work prior to its final submission. Finally, we are greatly appreciative of the anonymous reviewers for their time and constructive comments.
abstain abstain abstain objective objective abstain abstain abstain abstain other abstain result abstain method objective abstain other other other other other other abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain method abstain method abstain method method method method abstain abstain abstain abstain method method abstain other method method abstain result result abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain result objective method result other other other other