There is a growing body of work that proposes methods for mitigating bias in machine learning systems. These methods typically rely on access to protected attributes such as race, gender, or age. However, this raises two significant challenges: (1) protected attributes may not be available or it may not be legal to use them, and (2) it is often desirable to simultaneously consider multiple protected attributes, as well as their intersections. In the context of mitigating bias in occupation classification, we propose a method for discouraging correlation between the predicted probability of an individual's true occupation and a word embedding of their name. This method leverages the societal biases that are encoded in word embeddings, eliminating the need for access to protected attributes. Crucially, it only requires access to individuals' names at training time and not at deployment time. We evaluate two variations of our proposed method using a large-scale dataset of online biographies. We find that both variations simultaneously reduce race and gender biases, with almost no reduction in the classifier's overall true positive rate. In recent years, the performance of machine learning systems has improved substantially, leading to the widespread use of machine learning What's in a name? That which we call a rose by any in many domains, including high-stakes domains such as healthcare, employment, and criminal jus-tice (Chalfin et al., 2016; Miotto et al., 2017; Chouldechova, 2017). This increased prevalence has led many people to ask the question, accurate, but for whom? (Chouldechova and G'Sell, 2017). When the performance of a machine learning system differs substantially for different groups of people, a number of concerns arise (Baro-cas and Selbst, 2016; Kim, 2016). First and foremost, there is a risk that the deployment of such a method may harm already marginalized groups and widen existing inequalities. Recent work highlights this concern in the context of online recruiting and automated hiring (De-Arteaga et al., 2019). When predicting an individual's occupation from their online biography, the authors show that if occupation-specific gender gaps in true positive rates are correlated with existing gender imbalances in those occupations, then those imbalances will be compounded over time a phenomenon sometimes referred to as the leaky pipeline. Second, the correlations that lead to performance differences between groups are often irrelevant. For example, while an occupation classifier should predict a higher probability of software engineer if an individual's biography mentions coding experience, there is no good reason for it to predict a lower probability of software engineer if the biography also mentions softball. Prompted by such concerns about bias in machine learning systems, there is a growing body of work on fairness in machine learning. Some of the foundational papers in this area highlighted the limitations of trying to mitigate bias using methods that are unaware of protected attributes such as race, gender, or age (e.g., Dwork et al., 2012). As a result, subsequent work has primarily focused on introducing fairness constraints, defined in terms of protected attributes, that reduce incentives to rely on undesirable correlations (e.g., Hardt et al., 2016; Zhang et al., 2018). This approach is particularly useful if similar performance can be achieved by slightly different meansi.e., fairness constraints may aid in model selection if there are many near-optima. In practice, though, any approach that relies on protected attributes may stand at odds with antidiscrimination law, which limits the use of protected attributes in domains such as employment and education, even for the purpose of mitigating bias. And, in other domains, protected attributes are often not available (Holstein et al., 2019). Moreover, even when they are, it is usually desirable to simultaneously consider multiple protected attributes, as well as their intersections. For example, Buolamwini (2017) showed that commercial gender classifiers have higher error rates for women with darker skin tones than for either women or people with darker skin tones overall. We propose a method for reducing bias in machine learning classifiers without relying on protected attributes. In the context of occupation classification, this method discourages a classifier from learning a correlation between the predicted probability of an individual's occupation and a word embedding of their name. Intuitively, the probability of an individual's occupation should not depend on their namenor on any protected attributes that may be inferred from it. We present two variations of our methodi.e., two loss functions that enforce this constraintand show that they simultaneously reduce both race and gender biases with little reduction in classifier accuracy. Although we are motivated by the need to mitigate bias in online recruiting and automated hiring, our method can be applied in any domain where individuals' names are available at training time. Instead of relying on protected attributes, our method leverages the societal biases that are encoded in word embeddings (Bolukbasi et al., 2016; Caliskan et al., 2017). In particular, we build on the work of Swinger et al. (2019), which showed that word embeddings of names typically reflect the societal biases that are associated with those names, including race, gender, and age biases, as well encoding information about other factors that influence naming practices such as nationality and religion. By using word embeddings of names as a tool for mitigating bias, our method is conceptually simple and empirically powerful. Much like the proxy fairness approach of Gupta et al. (2018), it is applicable when protected attributes are not available; however, it additionally eliminates the need to specify which biases are to be mitigated, and allows simultaneous mitigation of multiple biases, including those that relate to group intersections. Moreover, our method only requires access to proxy information (i.e., names) at training time and not at deployment time, which avoids disparate treatment concerns and extends fairness gains to individuals with ambiguous names. For example, a method that explicitly or implicitly infers protected attributes from names at deployment time may fail to correctly infer that an individual named Alex is female and, in turn, fail to mitigate gender bias for her. Methodologically, our work is also similar to that of Zafar et al. (2017), which promotes fairness by requiring that the covariance between a protected attribute and a data point's distance from a classifier's decision boundary is smaller than some constant. However, unlike our method, it requires access to protected attributes, and does not facilitate simultaneous mitigation of multiple biases. Our method discourages an occupation classifier from learning a correlation between the predicted probability of an individual's occupation and a word embedding of their name. In this section, we present two variations of our methodi.e., two penalties that can be added to an arbitrary loss function and used when training any classifier. We assume that each data point corresponds to an individual, with a label indicating that individual's occupation. We also assume access to the names of the individuals represented in the training set. The first variation, which we call Cluster Constrained Loss (CluCL), uses k -means to cluster word embeddings of the names in the training set. Then, for each pair of clusters, it minimizes between-cluster disparities in the predicted probabilities of the true labels for the data points that correspond to the names in the clusters. In contrast, the second variation minimizes the covariance between the predicted probability of an individual's occupation and a word embedding of their name. Because this variation minimizes the covariance directly, we call it Covariance Constrained Loss (CoCL). The most salient difference between these variations is that CluCL only minimizes disparities between the latent groups captured by the clusters. For example, if the clusters correspond only to gender, then CluCL is only capable of mitigating gender bias. However, given a sufficiently large number of clusters, CluCL is able to simultaneously mitigate multiple biases, including those that relate to group intersections. For both variations, individual's names are not used as input to the classifier itself; they appear only in the loss function used when training the classifier. The resulting trained classifier can therefore be deployed without access to individuals' names. We define x i = { x 1 i , . . . , x Mi } to be a data point, y i to be its corresponding (true) label, and n fi and n li to be the first and last name of the corresponding individual. The classification task is then to (correctly) predict the label for each data point: p i = H ( x i ) (1)  y i = arg max 1  j | C | p i [ j ] , (2) where H (  ) is the classifier, C is the set of possible classes, p i  R | C | is the output of the classifier for data point x i e.g., p i [ j ] is the predicted probability of x i belonging to class j and  y i is the predicted label for x i . We define p y i to be the predicted probability of y i i.e., the true label for x i . The conventional way to train such a classifier is to minimize some loss function L , such as the cross-entropy loss function. Our method simply adds an additional penalty to this loss function: L total = L +   LCL , (3) where LCL is either L CluCL or L CoCL (defined in Sections 2.2 and 2.3, respectively), and  is a hyperparameter that determines the strength of the penalty. This loss function is only used during training, and plays no role in the resulting trained classifier. Moreover, it can be used in any standard setup for training a classifiere.g., training a deep neural network using mini-batches and the Adam optimization algorithm (Kingma and Ba, 2014). This variation represents each first name n fi and last name n li as a pair of low-dimensional vectors using a set of pretrained word embeddings E . These are then combined to form a single vector: n ei = 1 2 (cid:16) E [ n fi ] + E [ n li ] (cid:17) . Using k -means (Arthur and Vassilvitskii, 2007), CluCL then clusters the resulting embeddings into k clusters, yielding a cluster assignment k i for each name (and corresponding data point). Next, for each class c  C , CluCL computes the following average pairwise difference between clusters: l c = 1 k ( k  1)  k (cid:88) u,v =1   1 N c,u (cid:88) i : y i = c, k i = u p yi  1 N c,v (cid:88) i : y i = c, k i = v p yi   2 , (5) where u and v are clusters and N c,u is the number of data points in cluster u for which y i = c . CluCL considers each class individually because different classes will likely have different numbers of training data points and different disparities. Finally, CluCL computes the average of l 1 , . . . l | C | to yield L CluCL = 1 | C | (cid:88) c  C l c . This variation minimizes the covariance between the predicted probability of a data point's label and the corresponding individual's name. Like CluCL, CoCL represents each name as a single vector n ei and considers each class individually: l c = E i : y i = c (cid:2)(cid:0) p yi   cp (cid:1)  ( n ei   cn ) (cid:3) , (7) where  cp = E i : y i = c [ p yi ] and  cn = E i : y i = c [ n ei ] . Finally, CoCL computes the following average: L CoCL = 1 | C | (cid:88) c  C (cid:107) l c (cid:107) , where (cid:107)  (cid:107) is the (cid:96) 2 norm. One of our method's strengths is its ability to simultaneously mitigate multiple biases without access to protected attributes; however, this strength also poses a challenge for evaluation. We are unable to quantify this ability without access to these attributes. To facilitate evaluation, we focus on race and gender biases only because race and gender attributes are more readily available than attributes corresponding to other biases. We further conceptualize both race and gender to be binary (white/non-white and male/female) but note that these conceptualizations are unrealistic, reductive simplifications that fail to capture many aspects of race and gender, and erase anyone who does not fit within their assumptions. We emphasize that we use race and gender attributes only for evaluationthey do not play a role in our method. We use two datasets to evaluate our method: the adult income dataset from the UCI Machine Learning Repository (Dheeru and Karra Taniski-dou, 2017), where the task is to predict whether an individual earns more than $50k per year (i.e., whether their occupation is high status), and a dataset of online biographies (De-Arteaga et al., 2019), where the task is to predict an individual's occupation from the text of their online biography. Each data point in the Adult dataset consists of a set of binary, categorical, and continuous attributes, including race and gender. We preprocess these attributes to more easily allow us to understand the classifier's decisions. Specifically, we normalize continuous attributes to be in the range [0 , 1] and we convert categorical attributes into binary indicator variables. Because the data points do not have names associated with them, we generate synthetic first names using the race and gender attributes. First, we use the dataset of Tzioumis (2018) to identify white and non-white names. For each name, if the proportion of white people with that name is higher than 0.5, we deem the name to be white; otherwise, we deem it to be non-white. 1 Next, we use Social Security Administration data about baby names (2018) to identify male and female names. For each name, if the proportion of boys 1 For 90% of the names, the proportion of white people with that name is greater than 0.7 or less than 0.3, so there is a clear distinction between white and non-white names. with that name is higher than 0.5, we deem the name to be male; otherwise, we deem it to be female. 2 We then take the intersection of these two sets of names to yield a single set of names that is partitioned into four non-overlapping categories by (binary) race and gender. Finally, we generate a synthetic first name for each data point by sampling a name from the relevant category. Each data point in the Bios dataset consists of the text of an individual's biography, written in the third person. We represent each biography as a vector of length V , where V is the size of the vocabulary. Each element corresponds to a single word type and is equal to 1 if the biography contains that type (and 0 otherwise). We limit the size of the vocabulary by discarding the 10% most common word types, as well as any word types that occur fewer than twenty times. Unlike the Adult dataset, each data point has a name associated with it. And, because biographies are typically written in the third person and because pronouns are gendered in English, we can extract (likely) self-identified gender. We infer race for each data point by sampling from a Bernoulli distribution with probability equal to the average of the probability that an individual with that first name is white (from the dataset of Tzioumis (2018), using a threshold of 0.5, as described above) and the probability that an individual with that last name is white (from the dataset of Comenetz (2016), also using a threshold of 0.5). 3 Finally, like De-Arteaga et al. (2019), we consider two versions of the Bios dataset: one where first names and pronouns are available to the classifier and one where they are scrubbed. Throughout our evaluation, we use the fastText word embeddings, pretrained on Common Crawl data (Bojanowski et al., 2016), to represent names. Our method can be used with any classifier, including deep neural networks such as recurrent neural networks and convolutional neural networks. However, because the focus of this paper is mitigating bias, not maximizing classifier 2 For 98% of the names, the proportion of boys with that name is greater than 0.7 or less than 0.3, so there is an even clearer distinction between male and female names. 3 We note that, in general, an individual's race or gender should be directly reported by the individual in question; inferring race or gender can be both inaccurate and reductive. h i = W h  x i + b h p i = softmax ( h i ) where W h  R | C | M and b h  R | C | are the weights. This structure allows us to examine individual elements of the matrix W h in order to understand the classifier's decisions for any dataset. Both the Adult dataset and the Bios dataset have a strong class imbalance. We therefore use a weighted cross-entropy loss as L , with weights set to the values proposed by King and Zeng (2001). To quantify race bias and gender bias, we follow the approach proposed by De-Arteaga et al. (2019) and compute the true positive rate (TPR) race gap and the TPR gender gapi.e., the differences in the TPRs between races and between genders, respectivelyfor each occupation. The TPR race gap for occupation c is defined as follows: TPR r,c = P (cid:104)  Y = c | R = r, Y = c (cid:105) (8) Gap r,c = TPR r,c  TPR  r,c , (9) where r and  r are binary races,  Y and Y are random variables representing the predicted and true occupations for an individual, and R is a random variable representing that individual's race. Similarly, the TPR gender gap for occupation c is TPR g,c = P (cid:104)  Y = c | G = g, Y = c (cid:105) (10) Gap g,c = TPR g,c  TPR  g,c , (11) where g and  g are binary genders and G is a random variable representing an individual's gender. To obtain a single score that quantifies race bias, thus facilitating comparisons, we calculate the root mean square of the per-occupation TPR race gaps: Gap RMS r = (cid:115) 1 | C | (cid:88) c  C Gap 2 r,c . We obtain a single score that quantifies gender bias similarly. The motivation for using the root mean square instead of an average is that larger values have a larger effect and we are more interested in mitigating larger biases. Finally, to facilitate worst-case analyses, we calculate the maximum TPR race gap and the maximum TPR gender gap. We again emphasize that race and gender attributes are used only for evaluating our method. We first demonstrate that word embeddings of names encode information about race and gender. We then present the main results of our evaluation, before examining individual elements of the matrix W h in order to better understand our method. We cluster the names associated with the data points in the Bios dataset, represented as word embeddings, to verify that such embeddings indeed capture information about race and gender. We perform k -means clustering (using the k -means++ algorithm) with k = 12 clusters, and then plot the number of data points in each cluster that correspond to each (inferred) race and gender. Figures 1a and 1b depict these numbers, respectively. Clusters 1, 2, 4, 7, 8, and 12 contain mostly white names, while clusters 3, 5, and 9 contain mostly non-white names. Similarly, clusters 4 and 8 contain mostly female names, while cluster 2 contains mostly male names. The other clusters are more balanced by race and gender. Manual inspection of the clusters reveals that cluster 9 contains mostly Asian names, while cluster 8 indeed contains mostly female names. The names in cluster 2 are mostly white and male, while the names in cluster 4 are mostly white and female. This suggests that the clusters are capturing at least some intersections. Together these results demonstrate that word embeddings of names do indeed encode at least some information about race and gender, even when first and last names are combined into a single embedding vector. For a longer discussion of the societal biases reflected in word embeddings of names, we recommend the work of Swinger et al. (2019). The results of our evaluation using the Adult dataset are shown in Table 1. The task is to predict whether an individual earns more than $50k per year (i.e., whether their occupation is high status). Because the dataset has a strong class imbalance, we report the balanced TPRi.e., we compute the per-class TPR and then average over the classes. We experiment with different values of the hyperparameter  . When  = 0 , the method is equivalent to using the conventional weighted cross-entropy loss function. Larger values of  in-crease the strength of the penalty, but may lead to 1 2 3 4 5 6 7 8 9 10 11 12 Cluster 0 10000 20000 30000 40000 50000 60000 N u m b e r o f s a m p l e s White Not white Unknown a less accurate classifier. Using  = 0 leads to significant gender bias: the maximum TPR gender gap is 0.303. This means that the TPR is 30% higher for men than for women. We emphasize that this does not mean that the classifier is more likely to predict that a man earns more than $50k per year, but means that the classifier is more likely to correctly predict that a man earns more than $50k per year. Both variations of our method significantly reduce race and gender biases. With CluCL, the root mean square TPR race gap is reduced from 0.12 to 0.085, while the root mean square TPR gender gap is reduced from 0.299 to 0.25. These reductions in bias result in less than one percent decrease in the balanced TPR (79.5% is decreased to 79.3%). With CoCL, the race and gender biases are further reduced: the root mean square TPR race gap is reduced to 0.08, while the root mean square TPR gender gap is reduced to 0.163, with 0.5% decrease in the balanced TPR. We emphasize that although our proposed method significantly reduces race and gender biases, neither variation can completely eliminate them. In order to understand how different values of hyperparameter  influence the reduction in race and gender biases, we perform additional experiments using CoCL where we vary  from 0 to 10. Figure 2 depicts these results. Larger values of  indeed reduce race and gender biases; however, to achieve a root mean square TPR gender gap of zero means reducing the balanced TPR to 50%, which is unacceptably low. That said, there are a wide range of values of  that significantly reduce race and gender biases, while maintaining an acceptable balanced TPR. For example,  = 6 results in a root mean square TPR race gap of 0.038 and a root mean square TPR gender gap of 0.046, with only a 7.3% decrease in the balanced TPR. The results of our evaluation using the original and scrubbed (i.e., names and pronouns are scrubbed) versions of the Bios dataset are shown in Tables 2 and 3, respectively. The task is to predict an individual's occupation from the text of their online biography. Because the dataset has a strong class imbalance, we again report the balanced TPR. CluCL and CoCL reduce race and gender biases for both versions of the dataset. For the original version, CluCL reduces the root mean square TPR gender gap from 0.173 to 0.165 and the maximum TPR gender gap by 2.5%. Race bias is also reduced, though to a lesser extent. These reductions reduce the balanced TPR by 0.7%. For the scrubbed version, the reductions in race and gender biases are even smaller, likely because most of the information about race and gender has been removed by scrubbing names and pronouns. We hypothesize that these smaller reductions in race and gender biases, compared to the Adult dataset, are because the Adult dataset has fewer attributes and classes than the Bios dataset, and contains explicit race and gender information, making the task of reducing biases much simpler. We also note that each biography in the Bios dataset is represented as a vector of length V , where V is over 11,000. This means that the corresponding classifier has a very large number of weights, and there is a strong overfitting effect. Because this overfitting effect increases with  , we suspect it explains why CluCL has a larger root mean square TPR gender gap when  = 2 than when  = 1 . Indeed, the root mean square TPR gender gap for the training set is 0.05 when  = 2 . Using dropout and (cid:96) 2 weight regularization lessened this effect, but did not eliminate it entirely. Our method mitigates bias by making training-time adjustments to the classifier's weights that minimize the correlation between the predicted probability of an individual's occupation and a word embedding of their name. Because of our choice of classifier (a single-layer neural network, as described in Section 3.2), we can examine individual elements of the matrix W h to understand the effect of our method on the classifier's decisions. Figure 3a depicts the values of several weights for the conventional weighted cross-entropy loss function (i.e.,  = 0 ) and for CoCL with  = 2 for the Adult dataset. When  = 0 , the attributes sex Female and sex Male have large negative and positive weights, respectively. This means that the classifier is more likely to predict that a man earns more than $50k per year. With CoCL, these weights are much closer to zero. Similarly, the weights for the race attributes are also closer to zero. We note that the weight for age race_Black race_Whitesex_Female sex_Male 0.3 0.2 0.1 0.0 0.1 0.2 S c o r e s Regular CoCL, R=2  =2 the attribute age is also reduced, suggesting that CoCL may have mitigated some form of age bias. Figure 3b depicts the values of several weights specific to the occupation surgeon for the conventional weighted cross-entropy loss function (i.e.,  = 0 ) and for CoCL with  = 2 for the original version of the Bios dataset. When  = 0 , the attributes she and her have large negative weights, while the attribute he has a positive weight. This means that the classifier is less likely to predict that a biography that contains the words she or her belongs to a surgeon. With CoCL, these magnitudes of these weights are reduced, though these reductions are not as significant as the reductions shown for the Adult dataset. In this paper, we propose a method for reducing bias in machine learning classifiers without relying on protected attributes. In contrast to previous work, our method eliminates the need to specify which biases are to be mitigated, and allows simultaneous mitigation of multiple biases, including those that relate to group intersections. Our method leverages the societal biases that are encoded in word embeddings of names. Specifically, it discourages an occupation classifier from learning a correlation between the predicted probability of an individual's occupation and a word embedding of their name. We present two variations of our method, and evaluate them using a large-scale dataset of online biographies. We find that both variations simultaneously reduce race and gender biases, with almost no reduction in the classifier's overall true positive rate. Our method is conceptually simple and empirically powerful, and can be used with any classifier, including deep neural networks. Finally, although we focus on English, we expect our method will work well for other languages, but leave this direction for future work.
abstain abstain abstain objective abstain abstain objective result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain result objective method result method abstain objective abstain abstain abstain method method method method method abstain abstain method abstain abstain abstain abstain abstain method other method abstain method abstain abstain abstain abstain abstain other abstain abstain abstain other abstain method abstain method abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain method abstain method other abstain method method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective objective method abstain method result method method