A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or African Americans are athletic . Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. It is important to quantify to what extent these biases are present in them. Although this is a rapidly growing area of research, existing literature lacks in two important aspects: 1) they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natural data; 2) current evaluations focus on measuring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model. We address both these problems. We present StereoSet, a large-scale natural English dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. We contrast both stereotypical bias and language modeling ability of popular models like BERT, GPT2, ROBERTA , and XLNET . We show that these models exhibit strong stereotypical biases. Our data and code are available at https://stereoset. mit.edu . A key idea behind the current success of neural network models for language is pretrained representations such as word embeddings (Mikolov et al., 2013; Pennington et al., 2014) and pretrained language models (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019). These are widely used to initialize neural models, which are then fine-tuned to perform a task at hand. Typically, these are learned from massive text cor-Choose the appropriate word: Domain: Gender Target: Girl Context: Girls tend to be more ____ than boys Option 1: soft (stereotype) Option 2: determined (anti-stereotype) Option 3: fish (meaningless) (a) The Intrasentence Context Association Test Choose the appropriate sentence: Domain: Race Target: Arab Context: He is an Arab from the Middle East. Option 1: He is probably a terrorist with bombs. (stereotype) Option 2: He is a pacifist. (anti-stereotype) Option 3: My dog wants a walk. (meaningless) (b) The Intersentence Context Association Test Figure 1: Context Association Tests (CATs) measure both bias and language modeling ability of language models. pora using variants of language modeling objective (i.e., predicting a word given its surrounding context). In the recent years, these representations empowered neural models to attain unprecedented levels of performance gains on multiple language tasks. The resulting models are being deployed widely as services on platforms like Google Cloud and Amazon AWS to serve millions of users. While this growth is commendable, there are concerns about the fairness of these models. Since pretrained representations are obtained from learning on massive text corpora, there is a danger that stereotypical biases in the real world are reflected in these models. For example, GPT2 (Radford et al., 2019), a pretrained language model, has shown to generate unpleasant stereotypical text when prompted with context containing certain races such as African-Americans (Sheng et al., 2019). In this work, we assess the stereotypical biases of popular pretrained language models. The seminal works of Bolukbasi et al. (2016) and Caliskan et al. (2017) show that word embeddings such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) contain stereotypical biases using diagnostic methods like word analogies and association tests. For example, Caliskan et al. show that male names are more likely to be associated with career terms than female names where the association is measured using embedding similarity. Recently, studies have attempted to evaluate bias in contextual word embeddings where a word is provided with artificial context (May et al., 2019; Kurita et al., 2019), e.g., the contextual embedding of man is obtained from the embedding of man in the sentence This is a man . However, these have limitations. First, the context does not reflect the natural usage of a word. Second, they require stereotypical attribute terms to be predefined (e.g., pleasant and unpleasant terms). Third, they focus on single word terms and ignore multiword terms like construction worker . Lastly, they study bias of a model independent of its language modeling ability which could lead to undeserved trust in a model if it is a poor language model. In this work, we propose methods to evaluate stereotypical bias of pretrained language models. These methods do not have the aforementioned limitations. Specifically, we design two different association tests, one for measuring bias at sentence level ( intrasentence ), and the other at discourse level ( intersentence ) as shown in Figure 1.. In these tests, each target term (e.g., Arab) is provided with a natural context in which it appears, along with three possible associative contexts. The associative contexts help us to evaluate the biases of the model, as well as measure its language modeling performance. We crowdsource StereoSet , a dataset for associative contexts in English containing 4 target domains, 321 target terms and 16,995 test instances (triplets). Following previous literature (Greenwald and Banaji, 1995; Bolukbasi et al., 2016; Caliskan et al., 2017), we define a stereotype as an overgeneralized belief about a particular group of people, e.g., Asians are good at math . Our primary focus is on detecting the presence of stereotypes in pretrained language models. We leave the details of mitigating bias from pretrained language models to future work. We design our formulation around the desiderata of an ideal language model. An ideal language model should be able to perform the task of language modeling, i.e., it should rank meaningful contexts higher than meaningless contexts. For example, it should tell us that Our housekeeper is a Mexican is more probable than Our housekeeper is a banana . Second, it should not exhibit stereotypical bias, i.e., it should avoid ranking stereotypical contexts higher than anti-stereotypical contexts, e.g., Our housekeeper is a Mexican and Our housekeeper is an American should be equally possible. We desire equally possible instead of anti-stereotype over stereotype because any kind of overgeneralized belief is known to hurt target groups (Czopp et al., 2015). If the model consistently prefers stereotypes over anti-stereotypes, we say that the model exhibits stereotypical bias. Another approach would be to rank a neutral context higher over stereotypical or anti-stereotypical context. In practice, we found that collecting neutral contexts are prone to implicit biases and has low inter-annotator agreement (Section 4). Based on these observations, we develop the Context Association Test (CAT), a test that measures the language modeling ability as well as the stereotypical bias of pretrained language models. Although language modeling has standard evaluation metrics such as perplexity, due to varying vocabulary sizes of different pretrained models, this metric becomes incomparable across models. In order to analyse the relationship between language modeling ability and stereotypical bias, we define a simple metric that is appropriate for our task. Evaluating the full language modeling ability of models is beyond the scope of this work. In CAT, given a context containing a target group (e.g., housekeeper), we provide three different ways to instantiate this context. Each instantiation corresponds to either a stereotypical, anti-stereotypical, or a meaningless association. The stereotypical and anti-stereotypical associations are used to measure stereotypical bias, and the meaningless association is used to ensure that an unbiased language model still retains language modeling ability. We include the meaningless association in order to provide a standardized benchmark across both masked and autoregressive language models, which cannot be done with common metrics such as perplexity. Specifically, we design two types of association tests, intrasentence and intersentence CATs , to assess language modeling and stereotypical bias at sentence level and discourse level. Figure 1 shows an example for each. Our intrasentence task measures the bias and the language modeling ability at sentence-level. We create a fill-in-the-blank style context sentence describing the target group, and a set of three attributes, which correspond to a stereotype, an anti-stereotype, and a meaningless option (Figure 1a). In order to measure language modeling and stereotypical bias, we determine which attribute has the greatest likelihood of filling the blank, i.e., which of the instantiated contexts is more likely. Our intersentence task measures the bias and the language modeling ability at the discourse-level. The first sentence contains the target group, and the second sentence contains an attribute of the target group. Figure 1b shows the intersentence task. We create a context sentence with a target group that can be succeeded with three attribute sentences corresponding to a stereotype, an anti-stereotype and a meaningless option. We measure the bias and language modeling ability based on which attribute sentence is likely to follow the context sentence. Our work is inspired from related attempts that aim to measure bias in pretrained representations such as word embeddings and language models. The two popular methods of testing bias in word embeddings are word analogy tests and word association tests. In word analogy tests, given two words in a certain syntactic or semantic relation ( man  king ), the goal is generate a word that is in similar relation to a given word ( woman  queen ). Mikolov et al. (2013) showed that word embeddings capture syntactic and semantic word analogies, e.g., gender, morphology etc. Bolukbasi et al. (2016) build on this observation to study gender bias. They show that word embeddings capture several undesired gender biases (seman-tic relations) e.g. doctor : man :: woman : nurse . Manzini et al. (2019) extend this to show that word embeddings capture several stereotypical biases such as racial and religious biases. In the word embedding association test (WEAT, Caliskan et al. 2017), the association of two complementary classes of words, e.g., European and African names, with two other complementary classes of attributes that indicate bias, e.g., pleasant and unpleasant attributes, are studied to quantify the bias. The bias is defined as the difference in the degree with which European names are associated with pleasant and unpleasant attributes in comparison with African names being associated with those attributes. Here, the association is defined as the similarity between the name and attribute word embeddings. This is the first large scale study that showed word embeddings exhibit several stereotypical biases and not just gender bias. Our inspiration for CAT comes from WEAT. May et al. (2019) extend WEAT to sentence encoders, calling it the Sentence Encoder Association Test (SEAT). For a target term and its attribute, they create artificial sentences using generic context of the form "This is [target]." and "They are [attribute]." and obtain contextual word embeddings of the target and the attribute terms. They repeat Caliskan et al. (2017)'s study using these embeddings and cosine similarity as the association metric but their study was inconclusive. Later, Kurita et al. (2019) show that cosine similarity is not the best association metric and define a new association metric based on the probability of predicting an attribute given the target in generic sentential context, e.g., [target] is [mask] , where [mask] is the attribute. They show that similar observations of Caliskan et al. (2017) are observed on contextual word embeddings too. Our intrasentence CAT is similar to their setting but with natural context. We also go beyond intrasentence to propose intersentence CATs, since language modeling is not limited at sentence level. Concurrent to our work, Nangia et al. (2020) introduced CrowS-Pairs, which examines stereotypical bias via minimal pairs. However, CrowS-Pairs only studies bias within a single sentence (intrasentence) and ignores discourse-level (inter-sentence) measurements. Furthermore, StereoSet contains an order of magnitude of data that contains greater variety, and hence, has the potential to detect a wider range of biases that may be otherwise overlooked. Lastly, StereoSet measures bias across both masked and autoregressive language models, while CrowS-Pairs only measures bias in masked language models. Another method to evaluate bias in pretrained representations is to measure bias on extrinsic tasks like coreference resolution (Rudinger et al., 2018; Zhao et al., 2018) and sentiment analysis (Kir-itchenko and Mohammad, 2018). This method fine-tunes pretrained representations on the target task. The bias in pretrained representations is estimated by the target task's performance. However, it is hard to segregate the bias of task-specific training data from the pretrained representations. Our CATs are an intrinsic way to evaluate bias in pretrained models. In StereoSet, we select four domains as the target domains of interest for measuring bias: gender, profession, race and religion. For each domain, we select terms (e.g., Asian) that represent a social group. For collecting target term contexts and their associative contexts, we employ crowdworkers via Amazon Mechanical Turk. 1 We restrict ourselves to crowdworkers in USA since stereotypes could change based on the country. Table 1 shows the overall statistics of StereoSet. We also provide a full data statement in Section 9 (Bender and Friedman, 2018). We curate diverse set of target terms for the target domains using Wikidata relation triples (Vran-decic and Krtzsch, 2014). A Wikidata triple is of the form < subject, relation, object > (e.g., < Brad Pitt, P106, Actor > ). We collect all objects occurring with the relations P106 (profession), P172 (race), and P140 (religion) as the target terms. We manually filter terms that are either infrequent or too fine-grained ( assistant producer is merged with producer ). We collect gender terms from 1 Screenshots of our Mechanical Turk interface and details about task setup are available in the Section 9.6. Nosek et al. (2002). A list of target terms is available in Appendix A.1. In the intrasentence CAT, for each target term, a crowdworker writes attribute terms that correspond to stereotypical, anti-stereotypical and meaningless associations of the target term. Then, they provide a context sentence containing the target term. The context is a fill-in-the-blank sentence, where the blank can be filled either by the stereotype term or the anti-stereotype term but not the meaningless term. In the intersentence CAT, they first provide a sentence containing the target term. Then, they provide three associative sentences corresponding to stereotypical, anti-stereotypical and meaningless associations. These associative sentences are such that the stereotypical and the anti-stereotypical sentences can follow the target term sentence but the meaningless ones cannot follow the target term sentence. We also experimented with a variant that asked crowdworkers to provide a neutral association for the target term, but found that crowdworkers had significant trouble remaining neutral. In the validation step (next section), we found that many of these neutral associations are often classified as stereotype or anti-stereotype by multiple validators. We conjecture that attaining neutrality is hard is due to anchoring bias (Tversky and Kah-neman, 1974), i.e., stereotypical associations are easy to think and access and could implicitly affect crowdworkers to tilt towards them. Therefore, we discard the notion of neutrality. Some examples are shown in Appendix A.4. In order to ensure that stereotypes reflect common views, we validate the data collected in the above step with additional workers. For each context and its associations, we ask five validators to classify each association into a stereotype, an anti-stereotype or a meaningless association. We only retain CATs where at least three validators agree on the labels. 2 This filtering results in selecting 83% of the CATs, indicating that there is regularity in stereotypical views among the workers. Table 10 shows detailed agreement scores for 2 One can increase the quality of the data further by selecting examples where four or more workers agree upon. stereotypes computed using the average of annotator agreement per example. Are people prone to view stereotypes negatively? To answer this question, we classify stereotypes into positive and negative sentiment classes using a sentiment classifier (details in Appendix A.2). As evident in Table 2, people do not always associate stereotypes with negative associations (e.g., Asians are good at math has positive senti-ment). However, people associate stereotypes with relatively more negative associations than anti-stereotypes (41% vs. 33%). We also extract keywords in StereoSet to analyze which words are most commonly associated with target groups. We define a keyword as a word that is more frequent in StereoSet than the natural distribution of words (Kilgarriff, 2009; Jakubicek et al., 2013). Table 3 shows the top keywords of each domain. These keywords indicate that target terms in gender and race are associated with physical attributes such as beautiful , feminine , masculine , etc., professional terms are associated with behavioural attributes such as pushy, greedy, hardwork , etc., and religious terms are associated with belief attributes such as diety, forgiving, reborn , etc. This aligns with expectations and indicates that multiple annotators use similar attributes. We split StereoSet based on the target terms: 25% of the target terms and their instances for the development set and 75% for the test set. We ensure terms in the development set and test set are disjoint. We do not have a training set since this defeats the purpose of StereoSet, which is to measure the biases of pretrained language models (and not the models fine-tuned on StereoSet). Our desiderata of an ideal language model is that it excels at language modeling while not exhibiting stereotypical biases. In order to determine success at both these goals, we evaluate both language modeling and stereotypical bias of a given model. We pose both problems as ranking problems. Language Modeling Score ( lms ) In the language modeling case, given a target term context and two possible associations of the context, one meaningful and the other meaningless, the model has to rank the meaningful association higher than meaningless association. The meaningful association corresponds to either the stereotype or the anti-stereotype option. We define the language modeling score ( lms ) of a target term as the percentage of instances in which a language model prefers the meaningful over meaningless association. We define the overall lms of a dataset as the average lms of the target terms in the split. The lms of an ideal language model is 100, i.e., for every target term in a dataset, the model always prefers the meaningful association of the term. As discussed in Section 2.2, the goal of this metric is not to evaluate the full scale language modeling ability, but only to provide an reasonable metric that allows comparison between different models to analyze the relationship between language modeling ability and stereotypical bias. Stereotype Score ( ss ) Similarly, we define the stereotype score ( ss ) of a target term as the percentage of examples in which a model prefers a stereotypical association over an anti-stereotypical association. We define the overall ss of a dataset as the average ss of the target terms in the dataset. The ss of an ideal language model is 50, for every target term, the model prefers neither stereotypical associations nor anti-stereotypical associations. Idealized CAT Score ( icat ) StereoSet motivates a question around how practitioners should prefer models for real-world deployment. Just because a model has low stereotypical bias does not mean it is preferred over others. For example, although a random language model exhibits the lowest stereotypical bias ( ss = 50) it is the worst language model ( lms = 50). While model selection desiderata is often task-specific, we introduce a simple point-estimate called the idealized CAT ( icat ) score for model comparison assuming equal importance to language modeling ability and stereotypical bias. We define the icat score as lms  min ( ss, 100  ss ) 50 centered around the idea that an ideal language model has an icat score of 100 and a stereotyped model has a score of 0. Appendix A.6 presents a detailed formulation and Figure 2 (Appendix) highlights this idea. IDEALLM We define this hypothetical model as the one that always picks correct associations for a given target term context. It also picks equal number of stereotypical and anti-stereotypical associations over all the target terms. So the resulting lms and ss scores are 100 and 50 respectively. STEREOTYPEDLM We define this hypothetical model as the one that always picks a stereotypical association over an anti-stereotypical association. So its ss is 100 irrespective of its lms . RANDOMLM We define this model as the one that picks associations randomly, and therefore its lms and ss scores are both 50. SENTIMENTLM In Section 4.4, we saw that stereotypical instantiations are more frequently associated with negative sentiment than anti-stereotypes. In this baseline, we assess if sentiment can be used to detect a stereotypical association. For a given a pair of context associations, the model always picks the association with the most negative sentiment. In this section, we evaluate pretrained models such as BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), XLNET (Yang et al., 2019) and GPT2 (Radford et al., 2019) on StereoSet. While scoring sentences using autoregressive language models is well-defined, there is no corresponding scoring mechanism for masked language models. As a result, we evaluate our models using both likelihood-based scoring and psuedolikelihood scoring (Nangia et al., 2020). Likelihood-based Scoring For intrasentence CATs, we define the score as the log probability of an attribute term to fill the blank. If the attribute consists of multiple subwords, we iteratively unmask the subwords from left to right, and compute the average per-subword probability. We rank a given pair of attribute terms based on these probabilities (the one with higher probability is pre-ferred). In intersentence CATs, inspired by Devlin et al. (2019), we use a Next Sentence Prediction (NSP) task to rank the possible associations. For all models, we train identical Next Sentence Prediction heads on identical datasets (details given in Appendix A.5), and compute the log likelihood that any given target sentence follows the context. Given a pair of associations, we rank each association using this score. Psuedo-likelihood Scoring Nangia et al. (2020) adopts psuedo-likelihood based scoring (Salazar et al., 2020) that does not penalize less frequent attribute terms. In intrasentence CAT, we choose to never mask the attribute term but mask each context term one at a time and measure the psuedo-probability of the sentence given the attribute term. We refer the reader to Nangia et al. (2020) for more information on this scoring mechanism. In intersentence CATs, we measure the psuedolikelihood of the context sentence conditioned on the attribute sentence by iteratively masking the tokens in the context sentence while keeping the attribute sentence unchanged. Unlike above models, GPT2 is a generative model in an auto-regressive setting. For the intrasentence CAT, we instantiate the blank with an attribute term and compute the probability of the full sentence. Given a pair of associations, we rank each association using this score. For the intersentence CAT, our scoring mechanism mirrors that for masked language models. If the likelihood-based scoring mechanism is used, then we train an NSP head on identical datasets (details given in Appendix A.5) and compute the log likelihood that any given target sentence follows the context. If the masked language models are scored with psuedo-likelihood, then we measure the effect of the context sentence by measuring the joint probability of the attribute sentence with and without the context. Given a pair of associations, we rank each association by the ratio of these probabilities. Table 4 shows the overall results of baselines and models on StereoSet test set when using likelihood-based scoring, and Table 5 shows the results when using psuedo-likelihood based scoring. The results exhibit similar trends on the development and test sets. Since the initial version of this paper 3 used likelihood-based scoring, we mainly center the discussion around it as the trends are similar to pseudo-likelihood. Baselines vs. Models As seen in Table 4, all pretrained models have higher lms values than RANDOMLM indicating that these are better language models as expected. Among models, GPT2-large is the best performing language model (88.3) followed by GPT2-medium (85.9). Coming to stereotypical bias, all pretrained models demonstrate more stereotypical behavior than RANDOMLM. While GPT2-large is the most stereotypical model of all pretrained models (60.1), ROBERTA -base is the least stereotypical model (50.5). SENTIMENTLM achieves the highest stereotypical score compared to all pretrained models, indicating that sentiment can indeed be exploited to detect stereotypical associations. However, its language model performance is worse, which is expected, since sentiment alone isn't sufficient to distinguish meaningful and meaningless sentences. Relation between lms and ss All models exhibit a strong correlation between lms and ss (Spearman rank correlation  of 0.87). As the language model becomes stronger, its stereotypical bias ( ss ) does too. We build the strongest language model, ENSEMBLE , using a linear weighted combination of BERT-large, GPT2-medium, and GPT2-large, which is also found to be the most biased model ( ss = 62.5). The correlation between lms and ss is unfortunate and perhaps unModel LanguageModelScore( lms ) StereotypeScore( ss ) IdealizedCATScore( icat ) Test set IDEALLM 100 50.0 100 STEREOTYPEDLM 100 0.0 RANDOMLM 50.0 50.0 50.0 SENTIMENTLM 65.1 60.8 51.1 BERT-base 82.3 57.1 70.7 BERT-large 81.1 58.0 68.1 ROBERTA -base 83.5 58.5 69.4 ROBERTA -large 83.4 59.8 67.0 XLNET -base 60.5 52.4 57.6 XLNET -large 61.3 54.0 56.5 GPT2 86.8 59.0 71.1 GPT2-medium 88.6 61.6 68.0 GPT2-large 89.6 62.7 66.8 ENSEMBLE 90.1 62.2 68.1 Table 5: Performance of pretrained language models on the StereoSet test set, measured using psuedolikelihood scoring for the masked language models. avoidable as long as we rely on the real world distribution of corpora to train language models since these corpora are likely to reflect stereotypes. Amongst the models, GPT2 exhibits more unbiased behavior than other models ( icat score of 73.0). However, this metric is not intended as the sole criterion for model selection. Further research is required in designing better metrics. Impact of model size For a given architecture, all of its pretrained models are trained on the same corpora but with different number of parameters. For example, both BERT-base and BERT-large are trained on Wikipedia and BookCorpus (Zhu et al., 2015) with 110M and 340M parameters respectively. As the model size increases, we see that its language modeling ability ( lms ) increases, and correspondingly its stereotypical score. Impact of scoring mechanism We evaluate models using both likelihood based scoring and psuedo-likelihood based scoring. First, we note that likelihood-based ( ll ) scoring is higher than psuedo-likelihood-based ( pll ) scoring by a narrow margin (avg lms ll = 79 . 88 , avg lms pll = 79 . 68 ). For intrasentence CATs, psuedo-likelihood outperforms likelihood scoring by a wide margin (avg lms ll = 75 . 7 , avg lms pll = 79 . 4 ). However, psuedo-likelihood scoring is significantly degraded for intersentence CATs (avg lms ll = Model LanguageModelScore( lms ) StereotypeScore( ss ) IdealizedCATScore( icat ) Intrasentence Task BERT-base 82.5 57.5 70.2 BERT-large 82.9 57.6 70.3 ROBERTA -base 71.9 53.6 66.7 ROBERTA -large 72.7 54.4 66.3 XLNET -base 70.3 53.6 65.2 XLNET -large 74.0 51.8 71.3 GPT2 91.0 60.4 72.0 GPT2-medium 91.2 62.9 67.7 GPT2-large 91.8 63.9 66.2 ENSEMBLE 91.7 63.9 66.3 Intersentence Task BERT-base 88.3 61.7 67.6 BERT-large 88.7 60.6 71.0 ROBERTA -base 64.4 47.4 61.0 ROBERTA -large 78.8 55.2 70.6 XLNET -base 65.0 54.6 59.0 XLNET -large 82.5 56.1 72.5 GPT2 76.3 52.3 72.8 GPT2-medium 80.5 53.5 74.9 GPT2-large 84.9 56.1 74.5 ENSEMBLE 89.4 60.9 69.9 Table 6: Performance on the Intersentence and Intrasentence CATs on the StereoSet test set, measured using likelihood-based scoring. 78 . 82 , avg lms pll = 75 . 98 ). This suggests that psuedo-likelihood has trouble scoring longer sequences. Moreover, Aribandi et al. (2021) has shown that psuedo-likelihood has higher variance than likelihood scoring. Impact of pretraining corpora BERT, ROBERTA , XLNET and GPT2 are trained on 16GB, 160GB, 158GB and 40GB of text corpora. Surprisingly, the corpora size does not correlate with either lms or ss . This could be due to the differences in architectures and corpora types. A better way to verify this would be to train the same model on increasing amounts of corpora. Due to lack of computing resources, we leave this work for the community. We conjecture that the high performance of GPT2 (high lms and high ss ) is due to the nature of its training data. GPT2 is trained on documents linked from Reddit. Since Reddit has several subreddits related to target terms in StereoSet (e.g., relationships, religion), GPT2 is likely to be exposed to contextual Model LanguageModelScore( lms ) StereotypeScore( ss ) IdealizedCATScore( icat ) Intrasentence Task BERT-base 89.6 56.9 77.3 BERT-large 88.8 58.4 74.0 ROBERTA -base 88.0 58.5 73.0 ROBERTA -large 88.1 59.6 71.2 XLNET -base 60.6 51.3 59.0 XLNET -large 61.1 53.2 57.3 GPT2 91.0 60.4 72.0 GPT2-medium 91.2 62.9 67.7 GPT2-large 91.8 63.9 66.2 ENSEMBLE 91.9 63.9 66.3 Intersentence Task BERT-base 75.0 57.2 64.1 BERT-large 73.3 57.6 62.1 ROBERTA -base 79.1 58.4 65.9 ROBERTA -large 78.7 60.0 63.1 XLNET -base 60.4 53.5 56.2 XLNET -large 61.4 54.7 55.7 GPT2 82.5 57.6 70.0 GPT2-medium 85.9 60.3 68.3 GPT2-large 87.5 61.5 67.3 ENSEMBLE 89.1 61.1 69.9 Table 7: Performance on the Intersentence and Intrasentence CATs on the StereoSet test set, measured using psuedo-likelihood scoring. Domain-wise bias Table 8 shows domain-wise results of the ENSEMBLE model on the test set. The model is relatively less biased on race than on others ( ss = 61.8). We also show the most and least biased target terms for each domain from the development set (see Table 10 for human-agreement scores, a proxy for most and least biased terms). We conjecture that the most biased terms are those that have well established stereotypes and are also frequent in language. This is the case with mother (attributes: caring, cooking), software developer (attributes: geek, nerd), and Africa (attributes: poor, dark). The least biased are those that do not have well established stereotypes, for example, producer and Crimean . The outlier is Muslim , although it has established stereotypes indicated by the high human agreement (see Table 10). This requires further investigation. tence CATs on the test set. Since intersentence tasks has more number of words per instance, we expect intersentence language modeling task to be harder than intrasentence, especially results computed using psuedo-likelihood (Table 7). In this work, we develop the Context Association Test (CAT) to measure the stereotypical biases of pretrained language models in contrast with their language modeling ability. We crowdsource StereoSet , a dataset containing 16,995 CATs to test biases in four domains: gender, profession, race and religion. We show that current pretrained language models exhibit strong stereotypical biases. We also find that language modeling ability correlates with the degree of stereotypical bias. This dependence has to be broken if we are to achieve unbiased language models. We hope that StereoSet will spur further research in evaluating and mitigating bias in language models. We also note that achieving an ideal performance on StereoSet does not guarantee that a model is unbiased since bias can manifest in many ways (Gonen and Goldberg, 2019; Bender et al., 2021). We would like to thank the anonymous reviewers, Yonatan Belinkov, Vivek Kulkarni, and Spandana Gella for their helpful comments in reviewing this paper. This work was completed in part while MN and AB were at Intel AI.
abstain abstain abstain abstain other abstain method objective result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result abstain method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method method abstain method method method method abstain abstain method method objective other other other other other other other other other other abstain other other other other other abstain objective abstain other other other other other other other method method abstain method abstain other method method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method method method method method method abstain abstain method method abstain abstain method method abstain abstain abstain abstain method method abstain method abstain abstain method abstain method method method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective abstain result result result method result other other