Ryan Steed Carnegie Mellon University ryansteed@cmu.edu {swetasudha.panda,ari.kobren,michael.wick}@oracle.com A few large, homogenous, pre-trained models undergird many machine learning systems  and often, these models contain harmful stereotypes learned from the internet. We investigate the bias transfer hypothesis : the theory that social biases (such as stereotypes) internalized by large language models during pre-training transfer into harmful task-specific behavior after fine-tuning. For two classification tasks, we find that reducing intrinsic bias with controlled interventions before fine-tuning does little to mitigate the classifier's discriminatory behavior after fine-tuning. Regression analysis suggests that downstream disparities are better explained by biases in the fine-tuning dataset. Still, pre-training plays a role: simple alterations to co-occurrence rates in the fine-tuning dataset are ineffective when the model has been pre-trained. Our results encourage practitioners to focus more on dataset quality and context-specific harms. Large language models (LLMs) and other massively pre-trained foundation models are powerful tools for task-specific machine learning (Bom-masani et al., 2021). Models pre-trained by well-resourced organizations can easily adapt to a wide variety of downstream tasks in a process called fine-tuning . But massive pre-training datasets and increasingly homogeneous model design come with well-known, immediate social risks beyond the fi-nancial and environmental costs (Strubell et al., 2019; Bender et al., 2021). Transformer-based LLMs like BERT and GPT-3 contain quantifiable intrinsic social biases encoded in their embedding spaces (Goldfarb-Tarrant et al., 2021). These intrinsic biases are typically associated with representational harms, including stereotyping and denigration (Barocas et al., 2017; Blodgett et al., 2020; Bender et al., 2021). Separately, many studies document the extrinsic harms of the downstream (fine-tuned & task-specific) ap-(cid:37)(cid:68)(cid:86)(cid:72)(cid:3)(cid:48)(cid:82)(cid:71)(cid:72)(cid:79) (cid:11)(cid:72)(cid:17)(cid:74)(cid:17)(cid:3)(cid:53)(cid:82)(cid:37)(cid:40)(cid:53)(cid:55)(cid:68)(cid:12)(cid:3) (cid:51)(cid:85)(cid:72)(cid:16)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:72)(cid:71)(cid:3)(cid:48)(cid:82)(cid:71)(cid:72)(cid:79) (cid:41)(cid:76)(cid:81)(cid:72)(cid:16)(cid:55)(cid:88)(cid:81)(cid:72)(cid:71)(cid:3)(cid:48)(cid:82)(cid:71)(cid:72)(cid:79) (cid:51)(cid:85)(cid:72)(cid:16)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74) (cid:41)(cid:76)(cid:81)(cid:72)(cid:16)(cid:55)(cid:88)(cid:81)(cid:76)(cid:81)(cid:74) (cid:51)(cid:85)(cid:72)(cid:16)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:38)(cid:82)(cid:85)(cid:83)(cid:82)(cid:85)(cid:68) (cid:11)(cid:72)(cid:17)(cid:74)(cid:17)(cid:3)(cid:58)(cid:76)(cid:78)(cid:76)(cid:83)(cid:72)(cid:71)(cid:76)(cid:68)(cid:12)(cid:3) (cid:55)(cid:68)(cid:86)(cid:78)(cid:16)(cid:54)(cid:83)(cid:72)(cid:70)(cid:76)(cid:73)(cid:76)(cid:70)(cid:39)(cid:68)(cid:87)(cid:68)(cid:86)(cid:72)(cid:87)(cid:3)(cid:11)(cid:72)(cid:17)(cid:74)(cid:17)(cid:3)(cid:37)(cid:44)(cid:50)(cid:54)(cid:12)(cid:3) (cid:54)(cid:70)(cid:85)(cid:88)(cid:69)(cid:69)(cid:72)(cid:71)(cid:3)(cid:82)(cid:85)(cid:85)(cid:72)(cid:16)(cid:69)(cid:68)(cid:79)(cid:68)(cid:81)(cid:70)(cid:72)(cid:71)(cid:3) (cid:40)(cid:89)(cid:68)(cid:79)(cid:88)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81)(cid:55)(cid:72)(cid:80)(cid:83)(cid:79)(cid:68)(cid:87)(cid:72)(cid:86)(cid:3) (cid:48)(cid:72)(cid:68)(cid:86)(cid:88)(cid:85)(cid:72)(cid:3)(cid:44)(cid:81)(cid:87)(cid:85)(cid:76)(cid:81)(cid:86)(cid:76)(cid:70)(cid:3)(cid:37)(cid:76)(cid:68)(cid:86)(cid:11)(cid:72)(cid:17)(cid:74)(cid:17)(cid:3)(cid:83)(cid:85)(cid:82)(cid:81)(cid:82)(cid:88)(cid:81)(cid:3)(cid:85)(cid:68)(cid:81)(cid:78)(cid:76)(cid:81)(cid:74)(cid:12)(cid:3) (cid:48)(cid:72)(cid:68)(cid:86)(cid:88)(cid:85)(cid:72)(cid:3)(cid:40)(cid:91)(cid:87)(cid:85)(cid:76)(cid:81)(cid:86)(cid:76)(cid:70)(cid:3)(cid:37)(cid:76)(cid:68)(cid:86)(cid:11)(cid:72)(cid:17)(cid:74)(cid:17)(cid:3)(cid:55)(cid:51)(cid:53)(cid:3)(cid:74)(cid:68)(cid:83)(cid:12)(cid:3) (cid:51)(cid:72)(cid:85)(cid:87)(cid:88)(cid:85)(cid:69)(cid:72)(cid:71) (cid:53)(cid:68)(cid:81)(cid:71)(cid:82)(cid:80)(cid:76)(cid:93)(cid:72)(cid:71) (cid:39)(cid:72)(cid:16)(cid:69)(cid:76)(cid:68)(cid:86)(cid:72)(cid:71) (cid:56)(cid:83)(cid:86)(cid:87)(cid:85)(cid:72)(cid:68)(cid:80) (cid:39)(cid:82)(cid:90)(cid:81)(cid:86)(cid:87)(cid:85)(cid:72)(cid:68)(cid:80) (cid:37) (cid:76) (cid:68) (cid:86) (cid:3) (cid:55) (cid:85) (cid:68) (cid:81) (cid:86) (cid:73) (cid:72) (cid:85) Figure 1: Full pre-training to fine-tuning pipeline, with experimental interventions (green hexagons). plications of fine-tuned LLMs, including discriminatory medical diagnoses (Zhang et al., 2020), overreliance on binary gender for coreference resolution (Cao and Daum, 2021), the re-inforcement of traditional gender roles in part-of-speech tagging (Garimella et al., 2019), toxic text generation (Gehman et al., 2020), and censorship of inclusive language and AAVE (Blodgett and O'Connor, 2017; Blodgett et al., 2018; Park et al., 2018; Sap et al., 2019). Despite these risks, no research has investigated the extent to which downstream systems inherit social biases from pre-trained models. 1 Many stud-1 We use the term bias to refer to statistical associations 3524 ies warn that increasing intrinsic bias upstream may lead to an increased risk of downstream harms (Bolukbasi et al., 2016; Caliskan et al., 2017). This hypothesis, which we call the Bias Transfer Hypothesis , holds that stereotypes and other biased associations in a pre-trained model are transferred to post-fine-tuning downstream tasks, where they can cause further, task-specific harms. A weaker version of this hypothesis holds that downstream harms are at least mostly determined by the pre-trained model (Bommasani et al., 2021). In the pre-training paradigm, the extent to which the bias transfer hypothesis holds will determine the most effective strategies for responsible design. In the cases we study, reducing upstream bias does little to change downstream behavior. Still, there is hope: instead, developers can carefully curate the fine-tuning dataset, checking for harms in context. We test the bias transfer hypothesis on two classification tasks with previously demonstrated performance disparities: occupation classification of online biographies (De-Arteaga et al., 2019) and toxicity classification of Wikipedia Talks comments (Dixon et al., 2018). We investigate whether reducing or exacerbating intrinsic biases encoded by RoBERTa (Liu et al., 2019) decreases or increases the severity of downstream, extrinsic harms (Fig-ure 1). We find that the bias transfer hypothesis describes only part of the interplay between pretraining biases and harms after fine-tuning:  Systematically manipulating upstream bias has little impact on downstream disparity, especially for the most-harmed groups. With a regression analysis, we find that most variation in downstream bias can be explained by bias in the fine-tuning dataset (proxied by co-occurrence rates). Altering associations in the fine-tuning dataset can sometimes change downstream behavior, but only when the model is not pre-trained. Without absolving LLMs or their owners of representational harms intrinsic to pre-trained models, our results encourage practitioners and application stakeholders to focus more on dataset quality and context-specific harm identification and reduction. Little prior work directly tests the bias transfer hypothesis. The closest example of this phenomena is the blood diamond effect (Birhane and Prabhu, 2021), in which stereotyping and denigration in the pre-training corpora pervade subsequently generated images and language even before the fine-tuning stage (Steed and Caliskan, 2021). Still, it is unclear to what extent undesirable values encoded in pre-training datasets or benchmarkssuch as Wikipedia or ImageNet induce task-specific harms after fine-tuning (Baro-cas et al., 2019). Some work explores the consistency of intrinsic and extrinsic bias metrics: Goldfarb-Tarrant et al. (2021) find that intrinsic and extrinsic metrics are not reliably correlated for static embeddings like word2vec . We focus instead on state-of-the-art transformer-based LLMsthe subject of intense ethical debate (Bender et al., 2021; Bommasani et al., 2021)which construct contextual, rather than static, embeddings. Contextual embeddingstoken encodings that are conditional on other, nearby tokenspose an ongoing challenge for intrinsic bias measurement (May et al., 2019; Kurita et al., 2019; Guo and Caliskan, 2021) and bias mitigation (Liang et al., 2020). We find that intrinsic and extrinsic metrics are correlated for the typical LLMbut that the correlation is mostly explained by biases in the fine-tuning dataset. Other research tests the possibility that upstream mitigation could universally prevent downstream harm. Jin et al. (2021) show that an intermediate, bias-mitigating fine-tuning step can help reduce bias in later tasks. Likewise, Solaiman and Dennison (2021) propose fine-tuning on carefully curated values-targeted datasets to reduce toxic GPT-3 behavior. Our results tend to corroborate these methods: we find that the fine-tuning process can to some extent overwrite the biases present in the original pre-trained model. A recent post-hoc mitigation technique, on the other hand, debiases contextual embeddings before fine-tuning (Liang et al., 2020). Our results imply that while this type of debiasing may help with representational harms upstream, it is less successful for reducing harms downstream. To empirically evaluate the bias transfer hypothesis, we examine the relationship between upstream bias and downstream bias for two tasks. We track how this relationship changes under various controlled interventions on the model weights or the fine-tuning dataset. For each task, we fine-tune RoBERTa 2 (Liu et al., 2019). We split the fine-tuning dataset into train (80%), evaluation (10%), and test (20%) partitions. To fine-tune, we attach a sequence classification head and train for 3 epochs. 3 3.2 Occupation Classification The goal of occupation classification is to predict someone's occupation from their online biography. We fine-tune with the BIOS dataset scraped from Common Crawl (De-Arteaga et al., 2019), which includes over 400,000 online biographies belonging to 28 common occupations. Since self-identified gender was not collected, we will refer instead to the pronouns used in each biography (each biography uses either he/him or she/her pronouns). Following De-Arteaga et al. (2019), we use the scrubbed version of the datasetin which all the identifying pronouns have been removedto measure just the effects of proxy words (e.g. mother) and avoid overfitting on pronouns directly. Downstream Bias. Biographies with she/her pronouns are less frequently classified as belonging to certain traditionally male-dominated professionssuch as surgeonwhich could result in lower recruitment or callback rates for job candidates if the classifier is used by an employer. The empirical true positive rate (TPR) estimates the likelihood that the classifier correctly identifies a person's occupation from their biography. We follow previous work (De-Arteaga et al., 2019) in measuring downstream bias via the empirical true positive rate (TPR) gap between biographies using each set of pronouns. First, define TPR y,g = P [  Y = y | G = g, Y = y ] , where g is a set of pronouns and y is an occupation. Y and  Y represent the true and predicted occupation, respectively. Then the TPR bias (TPB) is TPB y = TPR y, she/her TPR y, he/him . (1) 2 roberta-base from HuggingFace (Wolf et al., 2020). 3 See Appendix D for more details. Epochs and other parameters were chosen to match prior work (Jin et al., 2021). For example, the classifier correctly predicts surgeon for he/him biographies much more often than for she/her biographies, so the TPR ratio for the surgeon occupation is low (see Appendix A). Upstream Bias. We adapt Kurita et al. (2019)'s pronoun ranking test to the 28 occupations in the BIOS dataset. Kurita et al. (2019) measure the encoded association of he/him and she/her pronouns by the difference in log probability scores between pronouns appearing in templates of the form {pronoun} is a(n) {occupation} . We augment this approach with 5 career-related templates proposed by Bartl et al. (2020) (see Appendix A). Formally, given a template sequence x y,g filled in with occupation y and pronoun g , we compute p y,g = P ( x y,g ) . As a baseline, we also mask the occupation and compute the prior probability  y,g = P ( x  y,g ) . The pronoun ranking bias (PRB) for this template is the difference in log probabilities: PRB y = log p y, she/her  y, she/her  log p y, he/him  y, he/him . For toxicity classification, we use the WIKI dataset, which consists of just under 130,000 comments from the online forum Wikipedia Talks Pages (Dixon et al., 2018). The goal of the task is to predict whether each comment is toxic. Each comment has been labeled as toxic or non-toxic by a human annotator, where a toxic comment is a rude, disrespectful, or unreasonable comment that is likely to make you leave the discussion (Dixon et al., 2018). Following Dixon et al. (2018), we focus on 50 terms referring to people of certain genders, races, ethnicities, sexualities, and religions. Downstream (Extrinsic) Bias. Mentions of certain identity groupssuch as queerare more likely to be flagged for toxic content, which could result in certain communities being systematically censored or left unprotected if an online platform uses the classifier. The classifier's empirical false positive rate (FPR) estimates its likelihood to falsely flag a non-toxic comment as toxic. The FPR corresponds to the risk of censoring inclusive speech or de-platforming individuals who often mention marginalized groups. Following Dixon et al. (2018), we express the classifier's bias against comments or commenters harmlessly mentioning an identity term as the FPR 3526 bias (FPB). Upstream Bias. Following Hutchinson et al. (2020), we measure upstream bias via sentiment associations. We construct a set of templates of the form {identity} {person} is [MASK] , where identities are the identity terms from Dixon et al. (2018) (e.g. gay or Muslim) and the person phrases include a person, my sibling, and other relations. We predict the top-20 likely tokens for the [MASK] position (e.g., awesome or dangerous). Using a pre-trained RoBERTA sentiment classifier trained on the TweetEval benchmark (Barbieri et al., 2020), we then measure the average negative sentiment score of the predicted tokens. The model's bias is the magnitude of negative association with each identity term. RoBERTa sometimes suggests terms which refer back to the target identity group. To mitigate this effect, we drop any predicted tokens that match the 50 identity terms (e.g. Latino) from Dixon et al. (2018), but we are likely missing other confounding adjectives (e.g. Spanish). We suspect this confounding is minimal: we achieve similar results with an alternative ranking-based bias metric (see Appendix C.2). No pre-training. To control for the effects of pre-training, we test randomly initialized versions of both models that have not been pre-trained. We average over 10 trials. Random perturbations. We instantiate a pre-trained model and then add random noise e to every weight in the embedding matrix. We try both uniform noise u  Unif (  c, c ) and Gaussian noise z  N (0 ,  2 ) , varying c and  2 . The final noise-added matrix is clipped so that its range does not exceed that of the original matrix. Bias mitigation. We apply the SENTDEBIAS algorithm to de-bias embeddings at the word-level (Liang et al., 2020). SENTDEBIAS estimates a bias subspace V with principal component analysis, then computes debiased word embeddings  h = h   P kj =1 h h , v j i v j by subtracting the projection of h onto V . We add the multiplier  to add or remove bias to various degrees standard SENTDEBIAS uses  = 1 . 0 . Re-balancing and scrubbing. For BIOS , we re-balance the fine-tuning dataset by undersampling biographies with the prevalent pronoun in each occupation. For WIKI , we randomly remove from the fine-tuning dataset  percent of comments mentioning each identity term. Our goal is to test the bias transfer hypothesis, which holds that upstream bias is transferred through fine-tuning to downstream models. By this view, we would expect changes to the pre-trained model to also change the distribution of downstream biasbut we find that for both tasks, downstream bias is largely invariant to upstream interventions. Figure 2 summarizes the similarity of biases before and after each randomized event. Though randomizing the model weights signifi-cantly reduces the mean and variance of upstream bias, the distribution of downstream bias changes very little. 4 For example, RoBERTa exhibits the same disparities in performance after fine-tuning regardless of whether the base model was pre-trained. Likewise, although the SENTDEBIAS mitigation method reduces pronoun ranking (upstream) bias as intended, we detect roughly the same downstream biases no matter the level of mitigation applied (Figure 3). For example, in the BIOS task, surgeons with he/him pronouns are still 1.3 times more likely to have their biographies correctly classified than their she/her counterparts. There is one notable exception to this trend: for the WIKI task, adding noise (uniform or Gaussian) to the pre-trained model's embedding matrix or not pre-training the model yields a modest reduction in median bias (Figure 2). As upstream bias shifts towards zero, downstream bias also moves marginally towards zero. Still, the largest biases (e.g., against the term gay) do not decrease and may even increase after randomization. 4 See Appendix B.2 for a full set of correlation tests. Though the results in the preceding section suggest that there is no clear or consistent correspondence between changes in upstream bias and changes in downstream bias, there is still a noticeable correlation between baseline upstream and downstream bias (Pearson's  = 0 . 43 , p = 0 . 022 for BIOS ,  = 0 . 59 , p < 10  5 for WIKI see Appendix A). There is an important third variable that helps explain this correlation: cultural artifacts ingrained in both the pre-training and fine-tuning datasets. 5 RoBERTa learns these artifacts through co-occurrences and other associations between words in both sets of corpora. 5 For example, cultural biases about which pronouns belong in which occupations are likely to pervade both the pretraining dataset (e.g., Wikipedia) and the fine-tuning dataset (internet biographies). for model treatment m , occupation y , and pronoun ranking template s . TPB is the TPR bias (down-stream bias) from Eq. 1; PRB is the pronoun ranking bias (upstream bias) from Eq. 2; f s and c m are dummy variables (for ordinary least squares) or fixed effects to capture heterogeneous effects between templates and models (such as variations in overall embedding quality). We control for statistical dataset bias with  , the prevalence of she/her biographies within each occupation y in the fine-tuning data. We find that the dataset bias in the fine-tuning stage explains most of the correlation between upstream and downstream bias. Under the strong bias transfer hypothesis, we would expect the coefficient on upstream bias  1 to be statistically signifi-cant and greater in magnitude than the coefficient  2 on our proxy for dataset bias. But for both tasks, 3528 DJ Model Nurse Surgeon DJ Model Nurse Surgeon Downstream Bias (TPR Ratio) Upstream Bias (Pronoun Ranking)  50  25  10  1 0 1 10 25 50 0 1 2  0.25 0.00 0.25 0.50 Mitigation Multiplier  B i a s ( s h e / h e r  h e / h i m ) Figure 3: Log TPR bias per occupation after scaled SENTDEBIAS on the BIOS task. the opposite is true: fine-tuning dataset bias has a larger effect than upstream bias. Figure 4 reports the coefficient estimates for these two variables. (See Appendix C.1 for all estimates, standard errors, assumptions and additional specifications.) In the BIOS task, a large decrease in upstream bias corresponds to a small but statistically signifi-cant increase in downstream bias. On average, a reduction of 0.3 to the log likelihood gapequivalent to the reduction in bias towards nurses after upstream mitigationcorresponds to a 0.5% increase in the TPR ratio. Almost all the downstream bias in the BIOS task is explained by dataset bias instead: a 10% increase in the prevalence of she/her pronouns within an occupation corresponds to a much larger 6.5% increase in the TPR ratio. In the WIKI task, upstream bias has a more noticeable effectbut the effect of dataset bias is still much larger. The regression takes the same form as Eq. 4, where downstream bias is FPR bias (Eq. 3), upstream bias is negative sentiment, and  i is the proportion of toxic mentions of identity i . We additionally control for the prevalence of each identity term and the average length of toxic mentions of each identity termlonger comments are less likely to result in erroneous screening (Ap-pendix C.1). As in the previous regression, dataset bias explains more of the variation in downstream bias than does upstream bias. On average, a large increase in average negative sentiment against a given identity term (e.g. 0.1, one standard deviation) corresponds to only a modest 3.7% increase in FPR. In comparison, only a 10% increase in the prevalence of toxic mentions of an identity corresponds to an even larger 6.3% increase in FPR. We also check that intrinsic downstream bias also changes due to fine-tuning. We measure intrinsic bias again after fine-tuning and regress on downstream intrinsic bias instead of downstream extrinsic bias (Eq. 4). The results are consistent: after controlling for the overall increase in log likelihood, the effect of upstream intrinsic bias on downstream intrinsic bias is explained almost entirely by fine-tuning dataset bias (Appendix C.1). Given the strong relation between our proxies for dataset bias and downstream bias, we test whether manipulating these proxies admits some control over downstream bias. For example, were the fine-tuning dataset to include exactly as many she/her nurse biographies as he/him, would the model still exhibit biased performance on that occupation? Our findings suggest not. No matter the amount of re-sampling, downstream bias remains relatively stable for pre-trained RoBERTa. The distributions of downstream bias with and without re-balancing are almost perfectly correlated (Pear-son's  = 0 . 94 , p < 0 . 01 see Appendix B.1). Though co-occurrence statistics help to explain downstream bias, they are still only proxies for dataset bias. Directly altering these statistics via re-sampling the dataset does not alter the sentence-level context in which the words are used. Based on this result, we also try completely removing mentions of identity terms. Scrubbing mentions of identity termsin all comments or only in toxic commentsappears to reduce bias only when the model is not pre-trained and all mentions of the term are scrubbed (Figure 5). For a pre-trained model trained on scrubbed data, a 10% decrease in mentions of an identity term corresponds to a 7.2% 3529           Fine  tuning dataset bias Prevalance of she/her Upstream bias Likelihood gap 0.00 0.25 0.50 0.75 Coefficient       All pre  trained N=6020 Pre  trained N=140 Noise added N=1400 Balanced N=1400 Not pre  trained N=2940 Bias  mitigated N=1820 decrease in FPR. We speculate that RoBERTa relies on its high quality feature embeddings to learn proxy biases about identity terms based on the way they are used in the pre-training corpora. For example, our model classifies a sequence containing only the term gay as toxic without any context. If a term like gay is often used pejoratively on the web, RoBERTa is likely to infer that sentences including gay are toxic even if the term never appears in the fine-tuning dataset. But when the upstream model is not pre-trained, the fine-tuned model has no such prejudices. In this case, removing all mentions of identity results in a distribution of bias entirely uncorrelated with the control (Pearson's  = 0 . 09 , p > 0 . 1 ). Notably, though, even a small number of mentions of an identity term like gay in the fine-tuning dataset are enough for a randomly initialized model to exhibit the same biases as the pre-trained model (Figure 5). Our approach comes with several limitations. First, our results may not generalize to all tasks especially non-classification tasksor all kinds of bias (e.g., bias against AAVE or non-English speak-ers). Also, while similar studies of bias have been successfully applied to vision transformers (Steed and Caliskan, 2021; Srinivasan and Uchino, 2021), our results may vary for substrates other than English language. Second, Goldfarb-Tarrant et al. (2021) conclude that the lack of correlation between intrinsic bias indicators and downstream bias is because some embedding bias metrics are unsuitable for measuring model bias. To ensure our intrinsic and extrinsic metrics measure the same construct, we chose upstream indicators that correlate with real-world occupation statistics (Caliskan et al., 2017; Kurita et al., 2019). Pronoun ranking in particular may be more reliable for transformer models than 3530 gay jewish gay jewish gay jewish gay jewish gay jewish gay jewish All Toxic None  4  2 0 2 4 Downstream Bias (Log FPR ratio) M e n ti on s S c r ubb e d Pre  trained Not pre  trained Figure 5: FPR gap (downstream bias) after scrubbing toxic mentions of identity terms from the WIKI fine-tuning dataset. other metrics (Silva et al., 2021). Still, downstream, annotator prejudices and other label biases could skew our extrinsic bias metrics as well (Davani et al., 2021). Third, there may be other explanations for the relationship between upstream and downstream bias: for example, decreasing the magnitude of upstream bias often requires a reduction in model accuracy, though we attempt to control for between-model variation with fixed effects and other controls. Alternate regression specifications included in Appendix C.1 show how our results change with the inclusion of controls. Our results offer several points of guidance to organizations training and distributing LLMs and the practitioners applying them: Attenuating downstream bias via upstream interventionsincluding embedding-space bias mitigationis mostly futile in the cases we study and may be fruitless in similar settings. For a typical pre-trained model trained for the tasks we study, the fine-tuning dataset plays a much larger role than upstream bias in determining downstream harms. Still, simply modulating co-occurrence statistics (e.g., by scrubbing harmful mentions of certain identities) is not sufficient. Task framing, design, and data quality are also very important for preventing harm. If a model is pre-trained, it may be more resistant to scrubbing, re-balancing, and other simple modulations of the fine-tuning dataset. But, our results also corroborate a nascent, somewhat optimistic view of pre-training bias. LLMs' intrinsic biases are harmful even before downstream applications, and correcting those biases is not guaranteed to prevent downstream harms. Increased emphasis on the role of fine-tuning dataset bias offers an opportunity for practitioners to shift to more careful, quality-focused and context-aware approach to NLP applications (Zhu et al., 2018; Scheuerman et al., 2021). This study navigates several difficult ethical issues in NLP ethics research. First, unlike prior work, we do not claim to measure gender biasesonly biases related to someone's choice of personal pronouns. However, our dataset is limited to the English he/him and she/her, so our results do not capture biases against other pronouns. Our study is also very Western-centric: we study only English models/datasets and test for biases considered normatively pressing in Western research. Second, our training data (including pre-training datasets), was almost entirely scraped from internet users without compensation or explicit consent. To avoid exploiting these users further, we only used already-scraped data and replicated already-existing classifiers, and we do not release these 3531 data or classifiers publicly. Finally, the models we trained exhibit toxic, offensive behavior. These models and datasets are intended only for studying bias and simulating harms and, as our results show, should not be deployed or applied to any other data except for this purpose. Thanks to Maria De-Arteaga and Benedikt Boeck-ing for assistance with BIOS data collection. Thanks also to the reviewers for their helpful comments and feedback.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain method method abstain abstain method abstain abstain abstain result result objective result other other other other method other abstain other other other abstain other abstain method abstain method method method method abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain method abstain abstain method abstain abstain abstain abstain method abstain method method method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain method abstain result method method abstain abstain abstain result abstain abstain abstain abstain abstain method abstain method method abstain other other