## Downstream bias mitigation is all you need

Arkadeep Baksi, Rahul Singh, and Tarun Joshi

Model Risk Management, Wells Fargo 1

{arkadeepbaksi.rik, rahul.singh3, tarun.joshi1}@wellsfargo.com

## Abstract

The  advent  of  transformer-based  architectures  and  large  language  models  (LLMs)  have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and other sources, there has been a major concern about harmful prejudices that may potentially be transferred from the data. In many applications, these pre-trained LLMs are fine-tuned on task specific datasets, which can further contribute to biases. This paper studies the extent of biases absorbed by LLMs during pre-training as well as task-specific behaviour after fine-tuning. We found that controlled  interventions  on  pre-trained  LLMs,  prior  to  fine-tuning,  have  minimal  effect  on lowering biases in classifiers. However, the biases present in domain-specific datasets play a much bigger role, and hence mitigating them at this stage has a bigger impact. While pretraining  does  matter,  but  after  the  model  has  been  pre-trained,  even  slight  changes  to  cooccurrence rates in the fine-tuning dataset has a significant effect on the bias of the model.

## 1. Introduction

The use of large language models (LLMs), together with the transformer-based architectures, have significantly advanced the performance of natural language processing (NLP) models for task-specific machine learning (Bommasani, et al. 2021). However, the utilization of extensive pre-training datasets pose recognizable and immediate social risks, in the form of potential harmful prejudices (Strubell, Ganesh and McCallum 2019), (Bender, et al. 2021). These biases are  commonly  linked  to  representational  issues,  including  stereotyping  and  denigration (Barocas,  Crawford,  et  al.  2017),  (Bender,  et  al.  2021). Additionally,  various  studies  like (Bommasani, et al. 2021) detail the harms stemming from the application of fine-tuned LLMs in  downstream  tasks,  including  discriminatory  medical  diagnoses  (Zhang,  et  al.  2020), overreliance on binary gender for coreference resolution (Cao and III 2021), reinforcement of traditional gender roles in part-of-speech tagging (Garimella, et al. 2019), generation of toxic text (Gehman, et al. 2020), and the censorship of inclusive language and AAVE (Park, Shin and Fung 2018), (Sap, et al. 2019). Researchers caution that the escalation of intrinsic bias in the pre-training phase may heighten the risk of downstream harms (Bolukbasi, et al. 2016), (Islam, Bryson and Narayanan 2016). This is called the bias transfer hypothesis , which says that stereotypes and other biased associations present in a pre-trained model are transferred to

1 The views expressed in the paper are those of the authors and do not represent the views of Wells Fargo.

subsequent post-fine-tuning tasks, potentially causing further task-specific harms. A weaker version of this hypothesis suggests that pre-trained models determine the downstream harms (Bommasani, et al. 2021). Typically, one refers to the bias in a pre-trained language model as the upstream bias and the bias in a fine-tuned model as downstream bias. There has been work in the literature to develop controls to reduce upstream bias, using a meticulous approach in curating the fine-tuning dataset, and scrutinizing for potential harms within the specific context. Our  investigations  into  the  scenarios  we  examine  reveal  that  mitigating  upstream  bias  has limited impact on altering downstream behaviour.

To test  the  bias  transfer  hypothesis,  we  conducted  experiments  on  two  classification  tasks known to demonstrate bias based performance disparities: i) occupation classification of online biographies  (De-Arteaga,  et  al.  2019)  and  ii)  toxicity  classification  of  Wikipedia  Talks comments  (Dixon,  et  al.  2018).  Our  objective  was  to  determine  whether  diminishing  or intensifying the intrinsic biases encoded into BERT (Devlin, et al. 2019) results in a reduction or escalation of downstream, extrinsic harms. Our findings reveal that:

- · The predominant variation in downstream bias can be accounted for by bias present in the fine-tuning dataset.
- · Modifying associations within the fine-tuning dataset occasionally induces changes in downstream behaviour, but this is observed only when the model is not pre-trained.

Based on these findings, we explored ways to modify the biases present in the fine-tuning dataset, primarily by identifying them in the first place when they are not explicitly mentioned. For this  we  utilized  the  bias  direction  encoded  in  the  embedding  space  to  get  a  bias  score associated with any text. Our results show the importance for practitioners to prioritize dataset quality and context-specific identification and reduction of harm.

## 2. Literature Review

There has been limited exploration of the bias transfer hypothesis in prior research. (Birhane and  Prabhu  2021)  showed  instances  where  stereotypes  and  denigration  from  pre-training corpora infiltrate generated images and language even prior to the fine-tuning phase. See also (Steed and Caliskan 2021). However, the degree to which undesirable values embedded in pretraining datasets or benchmarks, such as Wikipedia or ImageNet, lead to task-specific harms after fine-tuning remains uncertain (Barocas, Hardt and Narayanan 2019).

Our focus is on transformer based LLMs. These models produce contextual embeddings rather than static ones. Contextual embeddings rely on token encodings that are conditional on nearby tokens, and they pose an ongoing challenge for intrinsic bias measurement [(May, et al. 2019), (Kurita, et al. 2019), (Guo and Caliskan 2021)] and bias mitigation (Liang, et al. 2020). Our findings reveal a correlation between intrinsic and extrinsic metrics for typical LLMs, primarily stemming from biases in the fine-tuning dataset.

Additional research delves into the possibility that mitigating biases upstream could universally prevent downstream harm. (Jin, et al. 2021) demonstrated that an intermediary bias-mitigating fine-tuning step can alleviate bias in subsequent tasks. Similarly, (Dennison 2021) proposed fine-tuning on meticulously curated "values-targeted" datasets to counter toxic behaviour in GPT-3.  Our  results  support  these  findings,  highlighting  that  the  fine-tuning  process  can partially  overwrite  biases  inherent  in  the  original  pre-trained  model.  A  recent  post-hoc mitigation technique proposed debiasing contextual embeddings before fine-tuning (Liang, et al.  2020).  However,  our  findings  suggest  that  while  this  debiasing  method  may  address representational harms upstream, its efficacy in reducing downstream harms is limited.

## 3. Methods

To evaluate the bias transfer hypothesis, we performed empirical assessments by examining the relationships between upstream and downstream bias across two classification tasks. We closely observed the dynamics of this relationships through various controlled interventions, including  changing  the  model  embedding  weights  or  adjustments  made  to  the  fine-tuning dataset.

## 3.1 Models

For each task, we fine-tuned BERT base model (Devlin, et al. 2019) ,which has around 110M (Million) parameters. BERT-base model contains an encoder with 12 Transformer blocks, 12 self-attention heads, and the hidden size of 768. It takes an input of a sequence of no more than 512  tokens  and  outputs  the  representation  of  the  sequence.  The  sequence  has  one  or  two segments, of which the first token of the sequence is always [CLS] which contains the special classification embedding and another special token [SEP] is used for separating segments. For text classification tasks, BERT takes the final hidden state 'h' of the first token [CLS] as the representation of the whole sequence. A simple SoftMax classifier is then added to the top of BERT to predict the probability of a class label. We split the fine-tuning dataset into train (80%), evaluation (10%), and test (10%) partitions, and trained the model for 3 epochs for both the classification tasks (Appendix).

## 3.2 Occupation Classification

The objective of occupation classification is to predict an individual's occupation based on their online biography. For fine-tuning, we utilize the BIOS dataset. This dataset has biographical information extracted from Common Crawl (De-Arteaga, et al. 2019) and comprises over 400,000 online biographies across 28 most common occupations based on the BLS Standard Occupation Classification system. See Figure 1 for a comprehensive list of occupations and their corresponding prevalence in the dataset.

As self-identified gender information was not collected, we referenced the pronouns employed in each biography (he/him or she/her) as an alternative source of gender alias. Following the approach outlined by (De-Arteaga, et al. 2019), we employed the "scrubbed" version of the dataset, eliminating all identifying pronouns to solely measure the impact of proxy words (e.g., "mother") and prevent overfitting on pronouns directly.

Downstream bias : Biographies using she/her pronouns are less frequently categorized under certain  traditionally  male-dominated  professions,  such  as  "surgeon".  The  empirical  true positive  rate  (TPR)  serves  as  an  estimate  of  the  likelihood  that  the  classifier  accurately identifies  a  person's  occupation  from  their  biography.  Consistent  with  prior  research  (DeArteaga, et al. 2019), (Steed, Panda, et al. 2022) we gauged downstream bias by computing the empirical TPR gap between biographies utilizing each set of pronouns.

Define

$$T P R _ { y, g } = \, P [ \hat { Y } \, = \, y \, | \, G \, = \, g, Y \, = \, y ] & & \dots ( 1 )$$

where 𝑔 is a set of pronouns and 𝑦 is an occupation. 𝑌 and Y ̂ represent the true and predicted occupation, respectively. Then, the TPR bias is

𝑇𝑃𝐵

$$T P B _ { y } = \frac { T P R _ { y, s h e } } { T P R _ { y, h e } } \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

As an illustration, the classifier demonstrates a higher accuracy in predicting the occupation "surgeon" for he/him biographies compared to she/her biographies, resulting in a low True Positive Rate (TPR) ratio for the "surgeon" occupation.

Upstream bias : We adopted (Kurita, et al. 2019)'s pronoun ranking test (see Figure 2 ) to the 28  occupations  present  in  the  BIOS  dataset.  (Kurita,  et  al.  2019)  measured  the  encoded association of he/him and she/her pronouns by calculating the difference in log probability scores between pronouns found in templates of the form {pronoun} is a(n) {occupation}. We extended this method by incorporating 5 career-related templates proposed by (Bartl, Nissim and Gatt 2020) (see Table 1 ). Formally, for a template sequence 𝑥𝑦,𝑔 filled in with occupation y and pronoun g, we computed 𝑝𝑦,𝑔 =  𝑃(𝑥 𝑦,𝑔 ) . As a baseline, we also masked the occupation and computed the prior probability, 𝜋𝑦,𝑔 =  𝑃(𝑥𝑦,𝑔 ) π .  As shown below, the pronoun ranking bias (PRB) for this template is the difference in log probabilities:

$$P R B _ { \{ y \} } = \, l o g \frac { p _ { y, s h e } } { \pi _ { y, s h e } } - l o g \frac { p _ { y, h e } } { \pi _ { y, h e } } & & \dots \, ( 3 )$$

Figure 1 : Frequency of occupations in BIOS Dataset

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-5fa952f3-7196-4d4b-ac2c-8b0544d56b14.jpg)

## 3.3 Toxicity Classification

We used the WIKI dataset, comprising just under 130,000 comments sourced from the online forum Wikipedia Talks Pages (Dixon, et al. 2018). The task's objective is to predict whether each comment exhibits toxicity. Human annotators have labelled each comment as toxic or non-toxic, defining a toxic comment as one that is "rude, disrespectful, or unreasonable and likely  to  make  you  leave  the  discussion"  (Dixon,  et  al.  2018).  Following  the  methodology outlined by (Dixon, et al. 2018), we focused on 25 terms referring to individuals of specific genders, races, ethnicities, sexualities, and religions.

Downstream Bias :  Mentions of certain identity groups, such as "queer," are more prone to being  flagged  for  toxic  content.  This  could  lead  to  the  systematic  censorship  or  lack  of protection for specific communities if an online platform adopts the classifier. The classifier's empirical False Positive Rate (FPR) serves as an estimate of its likelihood to incorrectly flag a non-toxic comment as toxic. The FPR represents the risk of censoring inclusive speech or deplatforming individuals who frequently discuss marginalized groups.

Figure 2 :  Distribution of Pronoun Ranking (Upstream) Bias across occupations in BIOS task

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-e05f0151-72c0-4123-adbc-8fde8e5e5859.jpg)

Following  (Dixon,  et  al.  2018),  we  quantified  the  classifier's  bias  against  comments  or commenters that innocuously mention an identity term through the FPR bias (FPB) as

$$F P B _ { i } = \frac { P [ \hat { T } \, = \, 0 \, | \, I \, = \, i, T \, = \, 1 ] } { P [ \hat { T } \, = \, 0 \, | \, T \, = \, 1 ] } \,, & & \dots ( 4 )$$

where   is an identity term 𝑖 , 𝑇  =  1 if the comment was deemed toxic by a human annotator and T ̂ is the predicted class of a comment.

Upstream  Bias: We  adopted  the  approach  outlined  by  (Hutchinson,  et  al.  2020)  through sentiment associations. We formulated a set of templates structured as {identity} {person} is [MASK] (see Table 1 ), where identities correspond to the terms defined by (Dixon, et al. 2018) (e.g., "gay" or "Muslim"), and the person phrases encompass expressions like "people" and other relational terms. Our objective was to predict the top 20 tokens for the "[MASK]" position (e.g., "awesome" or "dangerous"). Leveraging a pre-trained BERT sentiment classifier, which was trained on the TweetEval benchmark (Barbieri, et al. 2020), we then calculate the average negative  sentiment  score  of  the  predicted  tokens.  The  model's  bias  was  determined  by  the magnitude of the negative association with each identity term.

To address instances where BERT suggests terms referring to the target identity group, we excluded any predicted tokens that match the 20 identity terms (e.g., "Latino") from (Dixon, et al. 2018)

Table 1 : Templates used for Bias Measurement

| Bias                | Templates                                         | Target   | Source                        |
|---------------------|---------------------------------------------------|----------|-------------------------------|
| Occupation- Pronoun | [TARGET] is a(n) [OCCUPATION]                     | he, she  | (Kurita, et al. 2019)         |
| Occupation- Pronoun | [TARGET] works as a(n) [OCCUPATION]               | he, she  | (Bartl, Nissim and Gatt 2020) |
| Occupation- Pronoun | [TARGET] wants to become a(n) [OCCUPATION]        | he, she  | (Bartl, Nissim and Gatt 2020) |
| Occupation- Pronoun | [TARGET] applied for the position of [OCCUPATION] | he, she  | (Bartl, Nissim and Gatt 2020) |
| Identity- Sentiment | [IDENTITY] [TARGET] are [MASK]                    | people   | (Dixon, et al. 2018)          |

## 4. Experiments

We measured changes in upstream and downstream bias  for the following interventions.

- · No Pre-Training -To mitigate the impact of pre-training, we conducted experiments using randomly initialized versions of both models, ensuring they have not undergone any pre-training. We averaged the results over 5 trials for robustness.
- · Perturbation of Model weights Additionally, we implemented random perturbations by introducing noise to every weight in the embedding matrix. We explored uniform noise  sampled  from  the  uniform  distribution 𝑢 ~ 𝑈𝑛𝑖𝑓(𝑐, 𝑐) .  The  parameter 𝑐 was varied, and the resulting matrix with added noise was clipped to ensure its range does not surpass that of the original matrix.
- · Re-sampling and scrubbing  For  the  BIOS  dataset,  re-balancing  involved  undersampling biographies based on the prevalent pronoun in each occupation, ensuring a more equitable representation.

For the WIKI dataset, scrubbing was implemented by randomly removing a certain percentage of  comments  that  mentioned  each  identity  term  from  the  fine-tuning  dataset.  This  process aimed to minimize biases related to specific identity mentions in the training data.

## 4.1 Downstream Bias is mostly invariant across Model Interventions

Our objective was to scrutinize the bias transfer hypothesis, which posits that biases originating upstream  are  conveyed  through  fine-tuning  to  downstream  models.  According  to  this, modifications to the pre-trained model should induce changes in the distribution of downstream bias. However, our findings (see Figure 10 , Figure 11 ) reveal that, for both tasks, downstream bias remained consistent despite upstream interventions. Our results mostly align with (Steed, Panda, et al. 2022), who perform similar experiments on the same tasks.

## 4.2 Most Downstream Bias is explained by the Fine-Tuning Step

While the preceding section's results indicate an absence of a clear or consistent association between alterations in upstream bias and changes in downstream bias, a noticeable dependence between baseline upstream and downstream bias persists. An essential factor contributing to this  is  a  third  variable:  cultural  artifacts  embedded  in  both  the  pre-training  and  fine-tuning datasets. In our case, these cultural artifacts come in the form of pronouns like 'he/she,' identity terms like 'gay,' 'queer,' 'transgender,' etc. Apart from pronouns, these artifacts also entail words which are reflective of gender. BERT acquires knowledge of these artifacts through cooccurrences and other associations among words present in both corpora sets.

To  validate  this  explanation,  we  performed  a  straightforward  regression  analysis  covering interventions and evaluation templates. This analysis aimed to explore the relationship between these variables and assess their impact on the observed biases. We estimated

$$\log T P B _ { m, y } = \, \beta _ { 0 } + \beta _ { 1 } P R B _ { m, y, s } + \beta _ { 2 } \pi _ { y } \quad \quad \dots ( 5 )$$

for model treatment 𝑚 , occupation 𝑦, and pronoun ranking template 𝑠 . TPB is the TPR bias (downstream bias) from above PRB is the pronoun ranking bias (upstream bias) from above We  measured  'dataset  bias'  with 𝜋 ,  the  prevalence  of  'she/her'  biographies  within  each occupation 𝑦 in the fine-tuning data.

Our analysis indicated that the "dataset bias" during the fine-tuning stage plays a pivotal role in  explaining the dependence between upstream and downstream bias. Under a strong bias transfer hypothesis, the coefficient on upstream bias ( 𝛽1 ) would be expected to be significant and larger in magnitude than the coefficient( 𝛽2 ) on our proxy for dataset bias. But our findings reveal the opposite pattern. Figure 3 presents the coefficient estimates for these two variables. For both tasks, the impact of fine-tuning dataset bias was more substantial than that of upstream bias.

(a) BIOS Dataset

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-c3024bbd-b378-4ebe-8b55-87b467f62bac.jpg)

(b) WIKI Dataset

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-a13e0b7b-0917-4312-9a21-bc07b7a0450e.jpg)

Figure 3 : (a) and (b) shows the variation of the coefficient values associated with the upstream and dataset bias. For both tasks, across different model interventions we find that the dataset bias is more significant in determining the downstream bias than the upstream bias.

The impact of upstream bias is more pronounced in the WIKI task. However, it is important to note  that  the  effect  of  dataset  bias  remains  larger.  The  regression  model  follows  the  same structure as above , where downstream bias is represented by FPR bias ( above ), upstream bias corresponds  to  the  negative  sentiment,  and 𝜋 𝑖 denotes  the  proportion  of  toxic  mentions  of identity  . Like the previous regression, our findings indicate that dataset bias explains a larger 𝑖 portion of the variation in downstream bias compared to upstream bias (see Figure 3 ). This underscores the significance of dataset-specific biases in influencing the observed biases during both the pre-training and fine-tuning stages.

## 4.3 Clever scrubbing reduces downstream bias, but re-sampling does not help

Considering the robust relationship between our proxies for dataset bias and downstream bias, we explored whether manipulating these proxies offers some control over downstream bias. For instance, we investigated whether  adjusting the fine-tuning dataset to include an equal number of she/her and he/him nurse biographies would result in unbiased performance for that occupation. However, our findings as shown in Figure 4 suggest otherwise. Regardless of the extent of re-sampling, downstream bias remains stable for pre-trained BERT. The distributions of downstream bias with and without re-balancing exhibit a perfect correlation.

Although the presence of these pronouns provides insights into explaining downstream bias, it is crucial to acknowledge that there are some words which are not pronouns but are reflective of gender, and they serve as proxies for dataset bias. Hence manipulating just, the statistics of pronouns  through  dataset  re-sampling  does  not  bring  about  changes  in  the  sentence-level context in which the words are employed. This highlights the limitations of solely relying on pronoun statistics to address underlying biases in the training data.

Building on this result, we experimented with completely removing mentions of identity terms for WIKI Dataset and scrubbing some proxy words from BIOS dataset. For BIOS dataset, we found that just scrubbing the direct pronouns was not enough as there were other words in the biographies which encoded the gender bias. To find those proxy words, we searched in the embedding space. We got the embedding representation for the entire BIOS vocabulary from BERT  by  concatenating  the  last  four  hidden  representations.  Then  we  brute  forced  an embedding search via cosine similarity to find those words from the vocabulary which were close to the direct pronouns like 'he' or 'she' and had a significant presence in the dataset. We scrubbed all the mentions of the top 75 words we obtained in the process and fine-tuned BERT on the scrubbed dataset. We also did a manual validation that the words we found are related to either of the gender classes. Figure 5 shows a notable shift from the trend we observed in case of simple re-sampling. Now we observe that the downstream bias after removal of proxy words centres more across the zero-value indicating notable reduction in bias across the model interventions.  For  the  Pre-trained  and  Uniform-noise  interventions,  the  downstream  bias reduced for every occupation. Intuitively when the model is not pre-trained the downstream bias is almost zero across occupations, primarily because all sources of bias are cut off.

In case of WIKI dataset, the process of scrubbing mentions of identity terms, whether across all comments or specifically in toxic comments, seemed to have an impact on reducing bias only when the model is not pre-trained, and all mentions of the term are scrubbed (refer to

Figure  6 ).  This  suggests  that  the  effectiveness  of  this  approach  may  be  influenced  by  the absence  of  pre-training  and  the  comprehensive  removal  of  identity  term  mentions.  We hypothesize that BERT relies on its high-quality feature embeddings to learn proxy biases regarding identity terms based on their usage patterns in the pre-training corpora. For instance, our model categorizes a sequence containing only the term "gay" as toxic, even without any contextual information. If a term like "gay" is frequently used pejoratively on the web, BERT is likely to infer that sentences containing "gay" are toxic, even if the term never appears in the fine-tuning dataset.

However,  when  the  upstream  model  is  not  pre-trained,  the  fine-tuned  model  lacks  such preconceptions. In this scenario, removing all mentions of identity resulted in a bias distribution that is entirely uncorrelated with the control ( Figure 6 ) . This suggests that the absence of pretraining allows for a more neutral and context-dependent learning of biases during fine-tuning. Interestingly, it is worth noting that even a small number of mentions of an identity term, such as  "gay,"  in  the  fine-tuning  dataset  are  sufficient  for  a  randomly  initialized  model  to demonstrate  the  same  biases  as  the  pre-trained  model.  This  observation  suggests  that  the presence of identity term mentions during fine-tuning, even in limited quantities, can lead to the replication of biases seen in the pre-trained model.

Figure 4 : Downstream bias distribution for the BIOS dataset with and without re-sampling. The plots clearly suggest that even on re-sampling and making equal gender representations across occupations, still the downstream bias remains mostly invariant

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-10a9513f-eb8f-4e3f-9b7f-ef3e33df8797.jpg)

Figure 5: Downstream bias distribution before and after removal of proxy word.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-793a9375-4bcf-4cc7-9401-dc777420799c.jpg)

1

Figure 6: Downstream bias across different scrubbed versions of the WIKI Dataset

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-06012675-e981-406d-9a38-f86df3e00b77.jpg)

## 4.4 Bias Directions as Proxies for Protected Attributes

So far, the results of our experiments covey that dataset bias is a major player in contributing towards downstream harms and we can control it by adapting means to mitigate the dataset bias. However, given this premise, the question of how we can identify dataset bias for any dataset becomes important. In (Bolukbasi, et al. 2016), the idea of existence of a semantic relation  direction in  the  embedding spaces is used to study the  biases present in the wordembedding and the extent to which they agree with human notion of gender stereotypes. More specifically, they focused on gender bias and leverage a set of gendered word pairs to define the gender direction by computing the top principal component of a set of ten gender pair difference vectors. Taking inspiration from this, we examined the efficacy of utilizing the bias directions inherent in the word embeddings to produce a proxy for the protected attribute being considered for fairness testing. For example, if binary gender is being tested (with the female group as the protected class), we look for assigning an aggregate femaleness score to each sentence (observation) based on the femaleness score of each of the constituent words.

With a given gender direction in the Word2vec space, following (Bolukbasi, et al. 2016), we defined the femaleness score of each word as the magnitude of the projected word-vector on the gender direction. More concretely, for a word 𝑤 with the word-vector 𝑣𝑤

$$\text{Token femaleness Score:} = \ < \nu _ { w }, d _ { g } > \quad \quad ( 6 )$$

where 𝑑𝑔 is the gender direction, defined as the top principal component of some gendered word pair differences. We implemented our method for inferring the gender referenced in the text on the BIOS dataset. Figure 7 shows that our token femaleness score captures the gender bias encoded across each profession of the BIOS dataset. We calculated the femaleness score at the sentence level for each bio in the data set.

To assign a femaleness score to each sentence, we simply take an average of the femaleness score of each of the words in the sentence. We divided the BIOS dataset into male and female genders using the femaleness score, without using the actual gender labels. We see that if we define a classification rule based on

$$\begin{cases} f e m a l e \colon s e n t e n c e \ f e m a l e n e s s \ s c o r e \, > \, 0 \\ \ m a l e \colon s e n t e n c e \ f e m a l e n e s s \ s c o r e \, < = \, 0 \end{cases}$$

then  the  resulting  accuracy  would  have  been 96% .  The  histogram  in Figure 8 shows  that aggregate femaleness scores is effective in discriminating the male and female class.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-a2b210cf-2986-41dd-9628-f1f8b03ce01c.jpg)

Femaleness Score for Professions

Figure 7 : Bar chart of femaleness scores for professions in the  Bios dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-19563312-bdf1-4b02-96cd-29b01a34bd98.jpg)

Figure 8: Histogram of femaleness score for the Bios dataset for male and female classes.

In the previous sections we found that for BIOS dataset, just scrubbing the direct pronouns was not enough as there were other words in the biographies which encoded the gender bias, and to find those words we searched in the embedding space. Now we employ femaleness score at a word/token level to find those proxy words which encoded gender bias. We scrubbed all words from the biographies for which the token femaleness score exceeded a certain threshold and fine-tuned the BERT-base model across different interventions on the scrubbed dataset. Our experimentations as shown in Figure 9 reveals that the distribution of reduction in downstream bias across occupations is remarkably like what we obtained when we employed searching the embedding  space  to  find  these  proxy  words.  Interestingly,  although  for  pre-trained  and uniform-noise the reduction in downstream bias is remarkably similar, but in the non-pretrained case, the reduction in downstream bias is more when femaleness score is used, proving its efficacy.

Figure 9 : Downstream bias distribution after removal of proxy words using embedding search and femaleness score

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-b5ac5c69-a783-494d-8d68-c67a9fb54e31.jpg)

## 5. Concluding remarks

Our  findings  provide  several  insights  for  organizations  engaged  in  training  and  deploying LLMs) as well as for practitioners utilizing them:

- · Downstream  Bias  Attenuation  via  Upstream  Interventions  -  Attempts  to  attenuate downstream  bias  through  upstream  interventions  appear  mostly  ineffective  in  the scenarios we examined and may yield limited results in similar contexts.
- · Role of Fine-Tuning Dataset - For a typical pre-trained model designed for the studied tasks, the fine-tuning dataset's influence is significantly greater than upstream bias in determining downstream harms.
- · Complexity  Beyond  Co-occurrence  Statistics  -  Modulating  co-occurrence  statistics, such as scrubbing harmful mentions of specific identities, is not a sufficient solution. In case of the BIOS dataset, we saw that on removal of the words acting as a proxy to the gender, the downstream bias reduced. So effective task framing, design, and data quality also play crucial roles in preventing harm.
- · Resistance to Modulations in Pre-trained Language Models - Pre-trained models may exhibit resistance to interventions like scrubbing, re-balancing, and other straightforward modifications of the fine-tuning dataset.

While our results align with an optimistic perspective on pre-training bias, we acknowledge that LLMs' intrinsic biases are present even before downstream.

Our findings are subject to several limitations:

- · Generalization to Tasks and Bias Types: The generalizability of our results to all tasks, particularly non-classification tasks, and to all types of bias (e.g., non-English speakers) may be limited. While similar bias studies have been successful with vision transformers, variations may occur for substrates other than the English language.
- · Intrinsic Bias Metric Suitability: (Goldfarb-Tarrant, et al. 2021) suggest that the lack of correlation between intrinsic bias indicators and downstream bias may stem from certain embedding bias metrics being unsuitable for measuring model bias. In our study, to  ensure  alignment  between  intrinsic  and  extrinsic  metrics,  we  selected  upstream indicators correlated with real-world occupation statistics. However, biases introduced by annotators and label biases in downstream tasks could still impact our extrinsic bias metrics.
- · Alternative Explanations for Relationship: There may be alternative explanations for  the  observed  relationship  between  upstream  and  downstream  bias.  For instance, reducing  the  magnitude  of  upstream  bias  might  necessitate  a  trade-off  with  model accuracy.

These limitations underscore the need for caution in generalizing our findings and highlight the complexity of addressing bias in different contexts and tasks.

## Acknowledgements

We would like to express our sincere gratitude to the reviewers especially Vijayan Nair for their invaluable feedback and insightful suggestions. Their thorough and meticulous reviews significantly contributed to improving the quality and clarity of this research work. We deeply appreciate their time, effort and expertise, which have greatly enhanced our work. Additionally, we  would  like  to  extend  our  heartfelt  thanks  to  Sudjianto  Agus,  whose  support  and encouragement made this research possible. He provided us the opportunity to explore this topic and his leadership have been instrumental in the successful completion of the project.

## References

| Barbieri, Francesco, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. 'TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification.' Edited by Trevor Cohn, Yulan He and Yang Liu. Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics. 1644-1650. doi:10.18653/v1/2020.findings-emnlp.148.                               |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Barocas, Solon, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. 'The problem with bias: Allocative versus representational harms in machine.' 9th Annual Conference of the Special Interest Group for Computing, Information and Society.                                                                                                                                                                                        |
| Barocas, Solon, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine Learning. MIT Press. https://fairmlbook.org/.                                                                                                                                                                                                                                                                                                            |
| Bartl, Marion, Malvina Nissim, and Albert Gatt. 2020. 'Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias.' Proceedings of the Second Workshop on Gender Bias in Natural Language Processing. Barcelona, Spain (Online): Association for Computational Linguistics. 1-16. https://aclanthology.org/2020.gebnlp-1.1.                                                                                          |
| Bender, Emily M., Timnit Gebru, Angelina McMillanMajor, and Shmargaret Shmitchell. 2021. 'On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?' Conference on Fairness, Accountability, and Transparency. New York, NY, USA: Association for Computing Machinery. 610-623.                                                                                                                                             |
| Birhane, Abeba, and Vinay Uday Prabhu. 2021. 'Large image datasets: A pyrrhic win for computer vision?' 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). Los Alamitos, CA, USA: IEEE Computer Society. 1536-1546. doi:10.1109/WACV48630.2021.00158.                                                                                                                                                               |
| Blodgett, Su Lin, Johnny Wei, and Brendan O'Connor. 2018. 'Twitter Universal Dependency Parsing for African-American and Mainstream American English.' Edited by Iryna Gurevych and Yusuke Miyao. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics. 1415-1425. Accessed July. doi:10.18653/v1/P18-1131. |
| Blodgett, Su Lin, and Brendan O'Connor. 2017. 'Racial Disparity in Natural Language Processing: {A} Case Study of Social Media African-American English.' CoRR abs/1707.00061. http://arxiv.org/abs/1707.00061.                                                                                                                                                                                                                         |
| Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. 'Language (Technology) is Power: A Critical Survey of 'Bias' in NLP.' Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. 5454-5476. Accessed July. doi:10.18653/v1/2020.acl-main.485.                                                                                 |
| Bolukbasi, Tolga, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. 'Man is to Computer Programmer as Woman is to Homemaker? Debiasing                                                                                                                                                                                                                                                                           |

| Word Embeddings.' Edited by D D Lee, MSugiyama, U V Luxburg, I Guyon and R Garnett. Advances in Neural Information Processing Systems. Curran Associates, Inc. 4349-4357.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, et al. 2021. 'On the Opportunities and Risks of Foundation Models.' CoRR abs/2108.07258. https://arxiv.org/abs/2108.07258.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Cao, Yang Trista, and Hal Daumé III. 2021. 'Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*.' Cambridge, MA: MIT Press. 615-661. Accessed November. doi:10.1162/coli_a_00413.                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| De-Arteaga, Maria, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. 'Bias in Bios: A Case Study of Semantic Representation Bias in a High- Stakes Setting.' Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19. New York, NY, USA: Association for Computing Machinery. 120-128.                                                                                                                                                                                                                                                                                      |
| Dennison, Irene Solaiman and Christy. 2021. 'Process for adapting language models to society (palms) with values-targeted datasets.' Pre-Proceedings of Advances in Neural Information Processing Systems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. 'BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding.' Edited by Jill Burstein, Christy Doran and Thamar Solorio. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics. 4171-4186. doi:10.18653/v1/N19-1423.                                                                                                                                                                                                |
| Dixon, Lucas, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. 'Measuring and Mitigating Unintended Bias in Text Classification.' Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. New Orleans LA, USA: ACM. 67-73.                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Garimella, Aparna, Carmen Banea, Dirk Hovy, and Rada Mihalcea. 2019. 'Women's Syntactic Resilience and Men's Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing.' Edited by Anna Korhonen, David Traum and Lluís Màrquez. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics. 3493-3498. Accessed July. doi:10.18653/v1/P19-1339. Gehman, Samuel, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. 'RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models.' Edited by Trevor Cohn, Yulan He and Yang Liu. Findings of the Association for |
| Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics. 3356-3369. doi:10.18653/v1/2020.findings-emnlp.301.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

| Goldfarb-Tarrant, Seraphina, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam. 2021. 'Intrinsic Bias Metrics Do Not Correlate with Application Bias.' Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics. 1926-1940.                                                                                             |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Guo, Wei, and Aylin Caliskan. 2021. 'Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases.' Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. New York, NY, USA: Association for Computing Machinery. 122-133.                                                                                                                                                                                        |
| Hutchinson, Ben, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. 'Social Biases in NLP Models as Barriers for Persons with Disabilities.' Edited by Dan Jurafsky, Joyce Chai, Natalie Schluter and Joel Tetreault. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. 5491-5501. doi:10.18653/v1/2020.acl-main.487.                                      |
| Islam, Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2016. 'Semantics derived automatically from language corpora necessarily contain human biases.' http://arxiv.org/abs/1608.07187.                                                                                                                                                                                                                                                                                             |
| Jin, Xisen, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. 'On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning.' Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics. 3770- 3783. doi:10.18653/v1/2021.naacl-main.296.                                          |
| Kurita, Keita, Nidhi Vyas, Ayush Pareek, Alan WBlack, and Yulia Tsvetkov. 2019. 'Measuring Bias in Contextualized Word Representations.' Edited by Marta R. Costa-jussà, Christian Hardmeier, Will Radford and Kellie Webster. Proceedings of the First Workshop on Gender Bias in Natural Language Processing. Florence, Italy: Association for Computational Linguistics. 166-172. doi:10.18653/v1/W19-3823.                                                                              |
| Liang, Paul Pu, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and LouisPhilippe Morency. 2020. 'Towards Debiasing Sentence Representations.' Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. 5502-5515.                                                                                                                                                                    |
| May, Chandler, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. 'On Measuring Social Biases in Sentence Encoders.' Edited by Jill Burstein, Christy Doran and Thamar Solorio. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics. 622-628. doi:10.18653/v1/N19-1063. |

Park, Ji Ho, Jamin Shin, and Pascale Fung. 2018. 'Reducing Gender Bias in Abusive Language Detection.' Edited by Ellen Riloff, David Chiang, Julia Hockenmaier and

Jun'ichi Tsujii. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics. 2799-2804. doi:10.18653/v1/D18-1302.

Sap, Maarten, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. 'The Risk of Racial Bias in Hate Speech Detection.' Edited by Anna Korhonen, David Traum and Lluís Màrquez. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics. 1668-1678. doi:10.18653/v1/P19-1163.

Steed, Ryan, and Aylin Caliskan. 2021. 'Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases.' Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21. New York, NY, USA: Association for Computing Machinery. 701-713.

Steed, Ryan, Swetasudha Panda, Ari Kobren, and Michael Wick. 2022. 'Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models.' Edited by Smaranda Muresan, Preslav Nakov and Aline Villavicencio. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics. 3524-3542. doi:10.18653/v1/2022.acl-long.247.

Strubell, Emma, Ananya Ganesh, and Andrew McCallum. 2019. 'Energy and Policy Considerations for Deep Learning in {NLP}.' Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics. 3645-3650. doi:10.18653/v1/P19-1355.

Zhang, Haoran, Amy X. Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020. 'Hurtful words: quantifying biases in clinical contextual word embeddings.' Proceedings of the ACM Conference on Health, Inference, and Learning. New York, NY, USA: Association for Computing Machinery. 110-120.

## 6. Appendix

Figure 10 : Upstream versus Downstream bias distribution across different model interventions averaged over 5 trails. In this case, for the downstream model BERT is fine-tuned on a version of BIOS dataset where the mentions of direct pronouns are scrubbed.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-c7eb0d57-1c11-4e59-ae43-94c4387f0c51.jpg)

Figure 11 : Upstream (Avg. Neg. Sentiment) versus Downstream (log FPR) bias distribution across different model interventions averaged over 5 trails for the WIKI Dataset. Despite changes to the upstream model intervention (left), the downstream bias distribution mostly remains stable.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00612v1-figure-132f5bf6-d1b4-4cc9-a968-8d40f4aec5e6.jpg)

Table 2 : BERT model hyperparameters for fine-tuning on BIOS and TOXICITY Dataset.

| Hyperparameters               | Value                               |
|-------------------------------|-------------------------------------|
| optimizer                     | adamw                               |
| lr_scheduler_type             | linear                              |
| adam_beta1                    | 0.9                                 |
| adam_beta2                    | 0.999                               |
| warmup_steps                  | 2400 (10% of total number of steps) |
| weight_decay                  | 0.01                                |
| train_batch_size              | 32                                  |
| eval_batch_size               | 32                                  |
| epochs                        | 3                                   |
| train, validation, test split | 70%, 10%, 20%                       |
Arkadeep Baksi Rahul Singh Tarun Joshi
Downstream bias mitigation is all you need
The advent of transformer-based architectures and large language models
(LLMs) have significantly advanced the performance of natural language
processing (NLP) models. Since these LLMs are trained on huge corpuses of data
from the web and other sources, there has been a major concern about harmful
prejudices that may potentially be transferred from the data. In many
applications, these pre-trained LLMs are fine-tuned on task specific datasets,
which can further contribute to biases. This paper studies the extent of biases
absorbed by LLMs during pre-training as well as task-specific behaviour after
fine-tuning. We found that controlled interventions on pre-trained LLMs, prior
to fine-tuning, have minimal effect on lowering biases in classifiers. However,
the biases present in domain-specific datasets play a much bigger role, and
hence mitigating them at this stage has a bigger impact. While pre-training
does matter, but after the model has been pre-trained, even slight changes to
co-occurrence rates in the fine-tuning dataset has a significant effect on the
bias of the model.