Recent work has shown pre-trained language models capture social biases from the large amounts of text they are trained on. This has attracted attention to developing techniques that mitigate such biases. In this work, we perform an empirical survey of five recently proposed bias mitigation techniques: Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebias. We quantify the effectiveness of each technique using three intrinsic bias benchmarks while also measuring the impact of these techniques on a model's language modeling ability, as well as its performance on downstream NLU tasks. We experimentally find that: (1) Self-Debias is the strongest debiasing technique, obtaining improved scores on all bias benchmarks; (2) Current debiasing techniques perform less consistently when mitigating non-gender biases; And (3) improvements on bias benchmarks such as StereoSet and CrowS-Pairs by using debiasing strategies are often accompanied by a decrease in language modeling ability, making it difficult to determine whether the bias mitigation was effective. 1 1 Introduction Large pre-trained language models have proven effective across a variety of tasks in natural language processing, often obtaining state of the art performance (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020). These models are typically trained on large amounts of text, originating from unmoderated sources, such as the internet. While the performance of these pre-trained models is remarkable, recent work has shown that they capture social biases from the data they are trained on (May et al. 2019; Kurita et al. 2019; Webster et al. 2020; Nangia et al. 2020; Nadeem 1 Our code is publicly available: https://github. com/mcgill-nlp/bias-bench . et al. 2021, inter alia ). Because of these findings, an increasing amount of research has focused on developing techniques to mitigate these biases (Liang et al., 2020; Ravfogel et al., 2020; Webster et al., 2020; Kaneko and Bollegala, 2021; Schick et al., 2021; Lauscher et al., 2021). However, the proposed techniques are often not investigated thoroughly. For instance, much work focuses only on mitigating gender bias despite pre-trained language models being plagued by other social biases (e.g., racial or religious bias). Additionally, the impact that debiasing has on both downstream task performance, as well as language modeling ability, is often not well explored. In this paper, we perform an empirical survey of the effectiveness of five recently proposed debiasing techniques for pre-trained language models: 2 Counterfactual Data Augmentation (CDA; Zmi-grod et al. 2019; Webster et al. 2020), Dropout (Webster et al., 2020), Iterative Nullspace Projection (INLP; Ravfogel et al. 2020), Self-Debias (Schick et al., 2021), and SentenceDebias (Liang et al., 2020). Following the taxonomy described by Blodgett et al. (2020), our work studies the effectiveness of these techniques in mitigating representational biases from pre-trained language models. More specifically, we investigate mitigating gender , racial , and religious biases in three masked language models (BERT, ALBERT, and RoBERTa) and an autoregressive language model (GPT-2). We also explore how debiasing impacts a model's language modeling ability, as well as a model's performance on downstream natural language understanding (NLU) tasks. Concretely, our paper aims to answer the following research questions: Q1 Which technique is most effective in mitigating bias? 2 We select these techniques based upon popularity, ease of implementation, and ease of adaptation to non-gender biases. Q2 Do these techniques worsen a model's language modeling ability? Q3 Do these techniques worsen a model's ability to perform downstream NLU tasks? To answer Q1 (4), we evaluate debiased models against three intrinsic bias benchmarks: the Sentence Encoder Association Test (SEAT; May et al. 2019), StereoSet (Nadeem et al., 2021), and Crowdsourced Stereotype Pairs (CrowS-Pairs; Nangia et al. 2020). Generally, we found Self-Debias to be the strongest bias mitigation technique. To answer Q2 (5) and Q3 (6), we evaluate debiased models against WikiText-2 (Merity et al., 2017) and the General Language Understanding Evaluation (GLUE; Wang and Cho 2019) benchmark. We found debiasing tends to worsen a model's language modeling ability. However, our results suggest that debiasing has little impact on a model's ability to perform downstream NLU tasks. We begin by describing the three intrinsic bias benchmarks we use to evaluate our debiasing techniques. We select these benchmarks as they can be used to measure not only gender bias, but also racial and religious bias in language models. Sentence Encoder Association Test (SEAT). We use SEAT (May et al., 2019) as our first intrinsic bias benchmark. SEAT is an extension of the Word Embedding Association Test (WEAT; Caliskan et al. 2017) to sentence-level representations. Below, we first describe WEAT. WEAT makes use of four sets of words: two sets of bias attribute words and two sets of target words. The attribute word sets characterize a type of bias. For example, the attribute word sets { man , he , him , ... } and { woman , she , her , ... } could be used for gender bias. The target word sets characterize particular concepts. For example, the target word sets { family , child , parent , ... } and { work , office , profession , ... } could be used to characterize the concepts of family and career , respectively. WEAT evaluates whether the representations for words from one particular attribute word set tend to be more closely associated with the representations for words from one particular target word set. For instance, if the representations for the female attribute words listed above tended to be more closely associated with the representations for the family target words, this may be indicative of bias within the word representations. Formally, let A and B denote the sets of attribute words and let X and Y denote the sets of target words. The SEAT test statistic is s ( X, Y, A, B ) = (cid:88) x  X s ( x, A, B )  (cid:88) y  Y s ( y, A, B ) where for a particular word w , s ( w, A, B ) is de-fined as the difference between w 's mean cosine similarity with the words from A and w 's mean cosine similarity with the words from B s ( w, A, B )= 1 | A | (cid:88) a  A cos( w, a )  1 | B | (cid:88) b  B cos( w, b ) . where  denotes the mean and  denotes the standard deviation. Here, an effect size closer to zero is indicative of a smaller degree of bias in the representations. To create a sentence-level version of WEAT (re-ferred to as SEAT), May et al. (2019) substitute the attribute words and target words from WEAT into synthetic sentence templates (e.g.,  this is a [WORD] ) to create a collection of sentences. Now, given sets of sentences containing attribute and target words, the WEAT test statistic can be computed using sentence-level representations obtained from a pre-trained language model. 3 We refer readers to Appendix A for a list of the SEAT tests we use to measure each type of bias in our work. We report the effect size for each SEAT test we evaluate. StereoSet. As our second intrinsic bias benchmark, we use StereoSet (Nadeem et al., 2021), a crowdsourced dataset for measuring four types of stereotypical bias in language models. Each StereoSet example consists of a context sentence, for example  our housekeeper is [MASK] , and a set of three candidate associations (completions) for that sentenceone being stereotypical, another being anti-stereotypical, and a third being 3 We use a permutation on the SEAT test statistic to compute the significance of association between the attribute word sets and the target word sets. We refer readers to the original work of Caliskan et al. (2017) for a complete description of this test. unrelated. 4 Using the example above, a stereotypical association might be  our housekeeper is Mexican , an anti-stereotypical association might be  our housekeeper is American , and an unrelated association might be  our housekeeper is computer . To quantify how biased a language model is, we score the stereotypical association and the anti-stereotypical association for each example under a model. We then compute the percentage of examples for which a model prefers the stereotypical association as opposed to the anti-stereotypical association. We define this percentage as the stereotype score of a model. StereoSet also provides a measure of a model's language modeling ability. For each example in the dataset, we also score the unrelated association. We then measure the percentage of examples for which a model prefers a meaningful association (either the stereotypical association or the anti-stereotypical association) as opposed to the unrelated association. We define this percentage as the language modeling score of a model. We evaluate our debiased models against the StereoSet test set. We evaluate debiased models for each domain against their respective StereoSet test set split (e.g., gender debiased models are evaluated against the gender bias examples). Crowdsourced Stereotype Pairs (CrowS-Pairs). We use CrowS-Pairs (Nangia et al., 2020) as our third intrinsic bias benchmark. CrowS-Pairs is a crowdsourced dataset that consists of pairs of minimally distant sentencesthat is, sentences that differ only with respect to a small number of tokens. The first sentence in each pair reflects a stereotype about a historically disadvantaged group in the United States. For example, the sentence  people who live in trailers are alcoholics  reflects a possible socioeconomic stereotype. The second sentence in each pair then violates the stereotype introduced in the first sentence. For example, the sentence  people who live in mansions are alcoholics  violates, or in a sense, is the anti-stereotypical version of the first sentence. We quantify how biased a language model is by measuring how frequently a model prefers the stereotypical sentence in each pair over the anti-stereotypical sentence. Nangia et al. (2020) originally proposed using pseudo-likelihood-based 4 We consider only the intrasentence task from StereoSet. scoring (Salazar et al., 2020) for CrowS-Pairs, however, recent work has suggested that pseudo-likelihood-based scoring may be subject to model calibration issues (Desai and Durrett, 2020; Jiang et al., 2020). Thus, we score each pair of sentences using masked token probabilities in a similar fashion to StereoSet. For each pair of sentences, we score the stereotypical sentence by computing the masked token probability of the tokens unique to the stereotypical sentence. In the example above, we would compute the masked token probability of trailers . We score each anti-stereotypical sentence in a similar fashion. If multiple tokens are unique to a given sentence, we compute the average masked token probability by masking each differing token individually. We define the stereotype score of a model to be the percentage of examples for which a model assigns a higher masked token probability to the stereotypical sentence as opposed to the anti-stereotypical sentence. Below, we describe the five debiasing techniques we evaluate in this work. We refer readers to Appendix C for additional experimental details on each debiasing technique. Counterfactual Data Augmentation (CDA). CDA (Zmigrod et al., 2019; Dinan et al., 2020a; Webster et al., 2020; Barikeri et al., 2021) is a data-based debiasing strategy often used to mitigate gender bias. Roughly, CDA involves re-balancing a corpus by swapping bias attribute words (e.g., he / she ) in a dataset. For example, to help mitigate gender bias, the sentence  the doctor went to the room and he grabbed the syringe  could be augmented to  the doctor went to the room and she grabbed the syringe . The re-balanced corpus is then often used for further training to debias a model. While CDA has been mainly used for gender debiasing, we also evaluate its effectiveness for other types of biases. For instance, we create CDA data for mitigating religious bias by swapping religious terms in a corpus, say church with mosque , to generate counterfactual examples. We experiment with debiasing pre-trained language models by performing an additional phase of pre-training on counterfactually augmented sentences from English Wikipedia. 5 5 We list the bias attribute words we make use of in our study in Appendix B. 1880 DROPOUT . Webster et al. (2020) investigate using dropout regularization (Srivastava et al., 2014) as a bias mitigation technique. They investigate increasing the dropout parameters for BERT and ALBERT's attention weights and hidden activations and performing an additional phase of pre-training. Experimentally, they find increased dropout regularization reduces gender bias within these models. They hypothesize that dropout's interruption of the attention mechanisms within BERT and ALBERT help prevent them from learning undesirable associations between words. We extend this study to other types of biases. Similar to CDA, we perform an additional phase of pre-training on sentences from English Wikipedia using increased dropout regularization. SELF-DEBIAS . Schick et al. (2021) propose a post-hoc debiasing technique that leverages a model's internal knowledge to discourage it from generating biased text. Informally, Schick et al. (2021) propose using hand-crafted prompts to first encourage a model to generate toxic text. For example, generation from an autoregressive model could be prompted with  The following text discriminates against people because of their gender.  Then, a second continuation that is non-discriminative can be generated from the model where the probabilities of tokens deemed likely under the first toxic generation are scaled down. Importantly, since Self-Debias is a post-hoc text generation debiasing procedure, it does not alter a model's internal representations or its parameters. Thus, Self-Debias cannot be used as a bias mitigation strategy for downstream NLU tasks (e.g., GLUE). Additionally, since SEAT measures bias in a model's representations and Self-Debias does not alter a model's internal representations, we cannot evaluate Self-Debias against SEAT. SENTENCEDEBIAS . Liang et al. (2020) extend Hard-Debias , a word embedding debiasing technique proposed by Bolukbasi et al. (2016) to sentence representations. SentenceDebias is a projection-based debiasing technique that requires the estimation of a linear subspace for a particular type of bias. Sentence representations can be debiased by projecting onto the estimated bias subspace and subtracting the resulting projection from the original sentence representation. for computing a bias subspace. First, they define a list of bias attribute words (e.g., he / she ). Second, they contextualize the bias attribute words into sentences. This is done by finding occurences of the bias attribute words in sentences within a text corpus. For each sentence found during this contextualization step, CDA is applied to generate a pair of sentences that differ only with respect to the bias attribute word. Finally, they estimate the bias subspace. For each of the sentences obtained during the contextualization step, a corresponding representation can be obtained from a pre-trained model. Principle Component Analysis (PCA; Abdi and Williams 2010) is then used to estimate the principle directions of variation of the resulting set of representations. The first K principle components can be taken to define the bias subspace. Iterative Nullspace Projection (INLP). Ravfogel et al. (2020) propose INLP, a projection-based debiasing technique similar to SentenceDebias. Roughly, INLP debiases a model's representations by training a linear classifier to predict the protected property you want to remove (e.g., gender) from the representations. Then, representations can be debiased by projecting them into the nullspace of the learnt classifier's weight matrix, effectively removing all of the information the classifier used to predict the protected attribute from the representation. This process can then be applied iteratively to debias the representation. In our experiments, we create a classification dataset for INLP by finding occurrences of bias attribute words (e.g., he / she ) in English Wikipedia. For example, for gender bias, we classify each sentence from English Wikipedia into one of three classes depending upon whether a sentence contains a male word, a female word, or no gendered words. To investigate which technique is most effective in mitigating bias ( Q1 ), we evaluate debiased BERT, ALBERT, RoBERTa, and GPT-2 models against SEAT, StereoSet, and CrowS-Pairs. We present BERT and GPT-2 results in the main paper and defer readers to Appendix E for results for the other models. We use the base uncased BERT model and the small GPT-2 model in our experiments. SEAT Results. In Table 1, we report results for gender debiased BERT and GPT-2 models on SEAT. For BERT, we find two of our four debiased models obtain lower average absolute effect sizes than the baseline model. In particular, INLP performs best on average across all six SEAT tests. Notably, INLP and SentenceDebias both obtain lower average absolute effect sizes than the baseline model while the CDA and Dropout models do not. Intuitively, this may be due to INLP and SentenceDebias taking a more aggressive approach to debiasing by attempting to remove all gender information from a model's representations. For GPT-2, our results are less encouraging. We find all of the debiased models obtain higher average absolute effect sizes than the baseline model. However, we note that SEAT fails to detect any statistically significant bias in the baseline model in any of the six SEAT tests to begin with. We argue, alongside others (Kurita et al., 2019; May et al., 2019), that SEAT's failure to detect bias in GPT-2 brings into question its reliability as a bias benchmark. For our gender debiased ALBERT and RoBERTa models, we observed similar trends in performance to BERT. We also use SEAT to evaluate racial and religious bias in our models. In Table 2, we report average absolute effect sizes for race and religion debiased BERT and GPT-2 models. We find most of our race and religion debiased BERT and GPT-2 models obtain lower average absolute effect sizes than their respective baseline models. These trends were less consistent in our ALBERT and RoBERTa models. For BERT, four of the five gender debiased models obtain lower stereotype scores than the baseline model. However, the race debiased models do not perform as consistently well. We note that for race, only two of the five debiased models obtain lower stereotype scores than the baseline model. Encouragingly, we find four of the five religion debiased BERT models obtain reduced stereotype scores. We observed similar trends to BERT in our ALBERT and RoBERTa results. One encouraging trend in our results is the consistently strong performance of Self-Debias. Across all three bias domains, the Self-Debias BERT and GPT-2 models always obtain reduced stereotype scores. Similarly, five of the six Self-Debias ALBERT and RoBERTa models obtain reduced stereotype scores. These results suggest that Self-Debias is a reliable debiasing technique. CrowS-Pairs Results. In Table 4, we report CrowS-Pairs results for BERT and GPT-2. Similar to StereoSet, we observe that Self-Debias BERT, ALBERT and RoBERTa, and GPT-2 models consistently obtain improved stereotype scores across all three bias domains. We also observe a large degree of variability in the performance of our debiasing techniques on CrowS-Pairs. For example, the GPT-2 religion SentenceDebias model obtains a stereotype score of 35 . 24 , an absolute difference of 27 . 62 points relative to the baseline model's score. We hypothesize that this large degree of variability is due to the small size of CrowS-Pairs (it is  14 th the size of the StereoSet test set). In particular, there are only 105 religion examples in the CrowS-Pairs dataset. Furthermore, Aribandi et al. (2021) demonstrated the relative instability of the performance of pre-trained language models, such as BERT, on CrowS-Pairs (and StereoSet) across different pre-training runs. Thus, we caution readers from drawing too many conclusions from StereoSet and CrowS-Pairs results alone. Do SEAT, StereoSet, and CrowS-Pairs Reliably Measure Bias? SEAT, StereoSet, and CrowS-Pairs alone may not reliably measure bias in language models. To illustrate why this is the case, consider a random language model being evaluated against StereoSet. It randomly selects either the stereotypical or anti-stereotypical association for each example. Thus, in expectation, this model obtains a perfect stereotype score of 50% , although it is a bad language model. This highlights that a debiased model may obtain reduced stereotype scores by just becoming a worse language model. Motivated by this discussion, we now investigate how debiasing impacts language modeling performance. To investigate how debiasing impacts language modeling ( Q2 ), we measure perplexities before and after debiasing each of our models on WikiText-2 (Merity et al., 2017). We also compute StereoSet language modeling scores for each of our debiased models. We discuss our findings below. of WikiText-2 for our experiments. Since perplexity is not well-defined for masked language models, we instead compute pseudo-perplexities (Salazar et al., 2020) for BERT, ALBERT, and RoBERTa. We compute the perplexities of the GPT-2 models normally. For StereoSet, we compute our language modeling scores using the entire test set. strong correlation (negative) between a model's perplexity on WikiText-2 and its StereoSet language modeling score. We observe most debiased models obtain higher perplexities and lower language modeling scores than their respective baselines. Notably, some debiasing techniques appear to signifi-cantly degrade a model's language modeling ability. For instance, the SentenceDebias GPT-2 model obtains a perplexity of 65 . 493 twice as large as the perplexity of the baseline GPT-2 model. However, there are some exceptions to this trend. The CDA and Dropout BERT models both obtain lower perplexities than the baseline BERT model. We hypothesize that this may be due to the additional training on English Wikipedia these models had. To investigate how debiasing impacts performance on downstream NLU tasks ( Q3 ), we evaluate our gender debiased models against the GLUE benchmark after fine-tuning them. We report the results for BERT and GPT-2 in Table 6. Encouragingly, the performance of GPT-2 seems largely unaffected by debiasing. In some cases, we in fact observe increased performance. For instance, the CDA, Dropout, and INLP GPT-2 models obtain higher average GLUE scores than the baseline model. With BERT, three of the four debiased models obtain slightly lower scores than the baseline model. Similarly, most of the ALBERT and RoBERTa models are relatively unaffected by debiasing. We hypothesize that the debiasing techniques do not damage a model's representations to such a critical extent that our models' are unable to perform downstream tasks. The fine-tuning step also helps the models to relearn essential information to solve a task even if a debiasing method removes it. Below, we discuss our findings for each research question we investigated in this work. We also discuss some of the limitations of our study. Q1: Which technique is most effective in mitigating bias? We found Self-Debias to be the strongest debiasing technique. Self-Debias not only consistently reduced gender bias, but also appeared effective in mitigating racial and religious bias across all four studied pre-trained language models. Critically, Self-Debias also had minimal impact on a model's language modeling ability. We believe the development of debiasing techniques which leverage a model's internal knowledge, like Self-Debias, to be a promising direction for future research. Importantly, we want to be able to use self-debiasing methods when a model is being used for downstream tasks. Q2: Do these techniques worsen a model's language modeling ability? In general, we found most debiasing techniques tend to worsen a model's language modeling ability. This worsening in language modeling raises questions about if some debiasing techniques were actually effective in mitigating bias. Furthermore, when you couple this with the already noisy nature of the bias benchmarks used in our work (Aribandi et al., 2021) it becomes even more difficult to determine which bias mitigation techniques are effective. Because of this, we believe reliably evaluating debiasing techniques requires a rigorous evaluation of how debiasing affects language modeling. Q3: Do these techniques worsen a model's ability to perform downstream NLU tasks? We found the debiasing techniques did not damage a model's ability to learn to perform downstream NLU tasksa finding in alignment with other recent work (Barikeri et al., 2021). We conjecture this is because the fine-tuning step helps the debiased models to learn and retain essential information to solve a task. 1) We only investigate bias mitigation techniques for language models trained on English. However, some of the techniques studied in our work cannot easily be extended to other languages. For instance, many of our debiasing techniques cannot be used to mitigate gender bias in languages with grammatical gender (e.g., French). 6 2) Our work is skewed towards North American social biases. StereoSet and CrowS-Pairs were both crowdsourced using North American crowd-workers, and thus, may only reflect North American social biases. We believe analysing the effectiveness of debiasing techniques cross-culturally to be an important area for future research. Furthermore, all of the bias benchmarks used in this work have only positive predictive power. For example, a perfect stereotype score of 50% on StereoSet does not indicate that a model is unbiased. 3) Many of our debiasing techniques make simplifying assumptions about bias. For example, for gender bias, most of our debiasing techniques assume a binary definition of gender. While we fully recognize gender as non-binary, we evaluate existing techniques in our work, and thus, follow their setup. Manzini et al. (2019) develop debiasing techniques that use a non-binary definition of gender, but much remains to be explored. Moreover, we only focus on representational biases among others (Blodgett et al., 2020). To the best of our knowledge, we have performed the first large scale evaluation of multiple debiasing 6 See Zhou et al. (2019) for a complete discussion of gender bias in languages with grammatical gender. techniques for pre-trained language models. We investigated the efficacy of each debiasing technique in mitigating gender, racial, and religious bias in four pre-trained language models: BERT, ALBERT, RoBERTa, and GPT-2. We used three intrinsic bias benchmarks to evaluate the effectiveness of each debiasing technique in mitigating bias and also investigated how debiasing impacts language modeling and downstream NLU task performance. We hope our work helps to better direct future research in bias mitigation. We thank the members of SR's research group for helpful feedback throughout the duration of this project. We would also like to thank Span-dana Gella for feedback on early drafts of this manuscript and Mat Pikuliak for finding a bug in our code. SR is supported by the Canada CIFAR AI Chairs program and the NSERC Discovery Grant program. NM is supported by an IVADO Excellence Scholarship. In this work, we used a binary definition of gender while investigating gender bias in pre-trained language models. While we fully recognize gender as non-binary, our survey closely follows the original methodology of the techniques explored in this work. We believe it will be critical for future research in gender bias to use a more fluid definition of gender and we are encouraged by early work in this direction (Manzini et al., 2019; Dinan et al., 2020b). Similarly, our work makes use of a narrow definition of religious and racial bias. We also note we do not investigate the extrinsic harm caused by any of the studied pre-trained language models, nor any potential reduction in harm by making use of any of our studied debiasing techniques. In other words, we do not investigate how biases in pre-trained language models effect humans in real-world settings. Finally, we highlight that all of the intrinsic bias benchmarks used in this work have only positive predictive power. In other words, they can identify models as biased, but cannot verify a model as unbiased. For example, a stereotype score of 50% on StereoSet or CrowS-Pairs is not indicative of an unbiased model. Additionally, recent work demonstrated the potential unreliability of the bias benchmarks used in this work (Blodgett et al., 2021). Because of this, we caution readers from making definitive claims about bias in pre-trained language models based on these benchmarks alone.
abstain abstain objective method result abstain abstain other abstain abstain abstain abstain other method objective objective objective abstain objective objective method result method result result method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method method abstain method method abstain abstain method abstain method method abstain abstain method method method method abstain method abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method method abstain result abstain method method abstain result abstain abstain abstain method abstain abstain method method abstain abstain method method abstain method objective abstain abstain objective objective abstain other other other other abstain abstain method abstain abstain abstain abstain abstain abstain abstain method