Abstract Text representation models are prone to exhibit a range of societal biases, reflecting the noncontrolled and biased nature of the underlying pretraining data, which consequently leads to severe ethical issues and even bias amplifica-tion. Recent work has predominantly focused on measuring and mitigating bias in pretrained language models. Surprisingly, the landscape of bias measurements and mitigation resources and methods for conversational language models is still very scarce: it is limited to only a few types of bias, artificially constructed resources, and completely ignores the impact that debiasing methods may have on the final performance in dialog tasks, e.g., conversational response generation. In this work, we present REDDITBIAS , the first conversational data set grounded in the actual human conversations from Reddit, allowing for bias measurement and mitigation across four important bias dimensions: gender , race , religion , and queerness . Further, we develop an evaluation framework which simultaneously 1) measures bias on the developed REDDITBIAS resource, and 2) evaluates model capability in dialog tasks after model debiasing. We use the evaluation framework to benchmark the widely used conversational DialoGPT model along with the adaptations of four debiasing methods. Our results indicate that DialoGPT is biased with respect to religious groups and that some debiasing techniques can remove this bias while preserving downstream task performance. Pretrained language models and their corresponding contextualized representation spaces (Peters et al., 2018; Devlin et al., 2019) have recently been shown to encode and amplify a range of stereotypical human biases (e.g., gender or racial biases) (Zhao et al., 2019; Basta et al., 2019; Liang et al., 2020a,b), much like their static embedding predecessors predecessors (Bolukbasi et al., 2016; Caliskan et al., 2017; Dev and Phillips, 2019; Gonen and Goldberg, 2019; Lauscher et al., 2020a, inter alia ). Having models that capture or even amplify human biases brings about further ethical challenges to the society (Henderson et al., 2018), since stereotyping minoritized groups is a representational harm that perpetuates societal inequalities and unfairness (Blodgett et al., 2020). Human biases are in all likelihood especially harmful if encoded in conversational AI systems, like the recent DialoGPT model (Zhang et al., 2020), which directly interact with humans, possibly even taking part in intimate and personal conversations (Utami et al., 2017). Given the increasing presence of dialog systems and chatbots in everyday life, the body of work that focuses on detecting and mitigating biases in conversational systems is surprisingly limited (Lee et al., 2019; Liu et al., 2020a,b; Dinan et al., 2020a,b), albeit some more research has recently emerged in the wider context of biases in general-purpose language generation models (Qian et al., 2019; Sheng et al., 2019; Nadeem et al., 2020; Yeo and Chen, 2020). Most of these efforts 1) focus on a single bias dimension (predominantly gender bias), 2) operate on artificial data (i.e., not real-world dialog interactions), and  with the isolated exception of Liu et al. (2020b)  3) completely neglect to analyze the potential effects of debiasing on model performance in dialog (sub-)tasks (e.g., dialog state tracking). In this work, we aim to close all these gaps by introducing REDDITBIAS , the first 'real-world' data set for measuring and mitigating biases in dialog models, together with an evaluation framework that couples bias measures with downstream evaluation on dialog tasks. Contributions. The contributions of this work are threefold: 1) we construct REDDITBIAS , a resource for multi-dimensional bias evaluation and mitigation dedicated to conversational AI. Unlike other bias evaluation resources, REDDITBIAS is created from real-world conversations collected from the popular online discussion platform Reddit and manually annotated for multiple societal bias dimensions: (i) religion , with two bias analysis subdimensions  ( Jews , Christians ) and ( Muslims , Christians ), (ii) race ( African , American ), (iii) gender ( female , male ), and (iv) queerness ( LGBTQ , straight ); 2) Along with the resource, we propose a dialog-oriented bias evaluation framework: it couples (i) a perplexity-based bias measure meant to quantify the amount of bias in generative language models with (ii) performance measures on two concrete downstream dialogue tasks  dialog state tracking (DST) and conversational response generation (CRG). Such a setup allows to test whether bias mitigation comes at the expense of deteriorated downstream dialog performance; 3) Finally, we adapt four bias mitigation methods from the literature and profile their debiasing and downstream effects on conversational language models with our evaluation framework. Acknowledging the conversational nature of REDDITBIAS , we resort to the recently proposed DialoGPT model (Zhang et al., 2020) for our comparative evaluation study. Our experimental results indicate that (i) DialoGPT is significantly biased along two (out of five) bias evaluation dimensions and (ii) that some of the employed debiasing methods (see 4) manage to reduce the bias, at the same time preserving DialoGPT's conversational capabilities. We release REDDITBIAS together with all code online at: https://github.com/umanlp/RedditBias . We first describe the process of REDDITBIAS creation, carried out in three steps: 1) creation of bias specifications for multiple bias dimensions, 2) retrieval of candidates for biased comments based on the bias specifications, and 3) manual annotation of candidate comments for the presence of bias. Unlike prior work, which mostly focuses on one or two bias dimensions, our study encompasses five types of bias from four dimensions: (1) religion (two different bias types), (2) race , (3) gender , and (4) queerness . To measure or mitigate a bias, one must first formalize (i.e., specify) it. To this end, we start from the concept of an explicit bias specification (Caliskan et al., 2017; Lauscher et al., 2020a): an explicit bias specification BE = ( T 1 , T 2 , A 1 , A 2 ) consists of two sets of target terms or phrases T 1 and T 2 between which a bias is expected to exist w.r.t. two sets of attribute terms or phrases A 1 , and A 2 . Further, we opt for bias specifications that reflect the inequality between groups in power, i.e., dominant groups, and discriminated groups, i.e., minoritized groups : 1 for each BE , the set T 1 consists of terms describing a minoritized group with (negative) stereotypical terms in A 1 , while T 2 consists of terms describing a dominant group with (positive) stereotypical terms in A 2 . We compile bias specifications as follows. The two target lists T 1 and T 2 are created by manually compiling small sets of near-synonymous expressions that unambiguously refer to the minoritized and dominant groups, respectively (e.g., for dimension religion and Muslims as the minoritized group, we compile T 1 = { muslims , arabs , islamic people , islam , islamic culture } ). We then collect the list A 1 of stereotypical negative descriptors by engaging with sociological literature relating to the minoritized groups (Welch, 2007; Shaw, 2012; Black, 2015). 2 Finally, we create the corresponding list A 2 of positive descriptors by looking for (loose) antonyms of expressions in A 1 (e.g., if Jewish people  T 1 are stereotypically greedy  A 1 , we would then place generous into A 2 ). Note that designing bias specifications is a crucial step in most of the current debiasing approaches and that there exists a trade-off between employing a bigger set of specification terms and keeping the bias specifications clean. In this work, we generally focus on smaller and more precise term sets. We show partial term lists from our bias specifications in Table 1 and provide the full lists in the Appendix. Starting from the compiled bias specifications, we next retrieve candidates for stereotypical comments from Reddit using the Pushshift API. 3 To this end, we generate query strings by coupling each term from the target set T 1 identifying the minoritized group with each term from the corresponding stereotypical attribute set A 1  this gives a query 1 We borrow the terminology (i.e., minoritized groups vs. dominant groups or groups in power ) from the feminist discourse (e.g., D'Ignazio and Klein, 2020) 2 For example, Welch (2007) lists stereotypical negatives such as violent , drug dealer , or prison as strongly associated with African Americans. set Q = T 1  A 1 . 4 We then run each query from Q against the API with a search period of 3 . 33 years. In a postprocessing step, we clean the retrieved data by removing URLs, user names, and extra white spaces and by lower-casing the comments. We retain only the retrieved comments that are shorter than 150 characters. In many cases we observed that, while comments as a whole are not biased, the part of the comment that connects t  T 1 and a  A 1 , if taken out of context, is biased (e.g., he just thinks all blacks are criminals  ). To capture more biased phrases, we also extract a narrower context of + /  7 tokens from the target term t  T 1 . We then annotate for bias both (1) the whole comment and (2) this narrower context window around the target term extracted from the comment (as a standalone text). 4 To increase the likelihood that retrieved comments do express the bias of interest, we couple T 1 terms with correct forms of the verb to be (e.g., jews are instead of jews or husband is instead of husband ), as such phrases are more likely to introduce a biased statement. (i.e., phrases). Human annotators then assign a binary label indicating if a negative stereotypical bias is expressed to each comment and each corresponding phrase. 5 After an initial training of the annotators, we first carried out a small calibration study during which we refined the annotation guidelines 6 and identified corner cases, e.g., comments involving sarcasm or comments quoting an earlier (biased) comment. We then split all the retrieved candidate comments for all five bias types between the three annotators (without overlap) and let them carry out the annotation work. Table 3 reveals the total number of annotated and positive (i.e., biased) instances at the comment and phrase level for each of the five bias types. Finally, we measure the inter-annotator agreement (IAA) by letting an additional annotator 7 label 100 randomly selected candidates for biased comments (20 per each of the five bias types). We measure an IAA of .65 Krippendorff's  (nomi-nal) on the comment level and .67 on the phrase 5 We hired three annotators with diverse gender and diverse religious and cultural backgrounds; they all have an University degree in Computer Science and speak English fluently. level. We did not observe significant differences in agreement across the individual bias types. For the purposes of training and evaluating bias mitigation methods (which we adapt from the literature for conversational LMs in 4), we split the obtained biased phrases into train, development, and test portions; their sizes are also shown in Table 3. We further show examples of comments labeled as biased for all five bias types in Table 2. We now describe our framework for bias evaluation in conversational language models (LMs), which couples (1) a bias measure computed on the test portions of REDDITBIAS with (2) task-specific performance on downstream dialog tasks. The latter aims to capture potential negative effects that debiasing techniques may have on downstream dialog performance of conversational LMs. We estimate bias in conversational LMs by measuring if (and how much) likelier the LM is to generate a stereotypically biased phrase compared to a corresponding inversely biased phrase in which we replace t 1  T 1 with a t 2  T 2 . To this end, we start from a bias specification BE = ( T 1 , T 2 , A 1 , A 2 ) and a set of the corresponding biased phrases X ( T 1 ,A 1 ) from the test portion of REDDITBIAS related to this bias dimension. We first build pairs of corresponding terms between the { t 1 , t 2 }  T 1  T 2 . 8 We list all pairs in the Appendix. We then follow the principle of counterfactual data augmentation (Zhao et al., 2018) and for each biased phrase x ( t 1 ,a 1 )  X ( T 1 ,A 1) (e.g., everyone knows jews are greedy) create a corresponding inversely biased phrase  x ( t 2 ,a 1 ) (e.g., everyone knows christians are greedy). Let ( X ( T 1 ,A 1 ) ,  X ( T 2 ,A 1 ) ) = { ( x ( i ) ( t 1 ,a 1 ) ,  x ( i ) ( t 2 ,a 1 ) ) } Ni =1 be 8 For instance, for the bias type Religion #1 , we pair ( jew , christian ), ( judaism , christianity ), etc. a set of N such counterfactual pairs. Our bias measure relies on the significance of mean perplexity differences between biased expressions x ( i ) ( t 1 ,a 1 ) and their counterfactual counterparts  x ( i ) ( t 2 ,a 1 ) . Since the reliability of such significance may be negatively affected by outliers (Pollet and van der Meij, 2017), we first reduce noise by removing pairs in which either x ( i ) ( t 1 ,a 1 ) or  x ( i ) ( t 2 ,a 1 ) have very high perplexity, i.e., if they are not within the interval  [( x + 3  s ) , ( x  3  s )] , where  x is the mean perplexity of the sample and s the corresponding standard deviation. Finally, we quantify and report the bias effect as the t -value of the Student's two-tailed test between two ordered sets of corresponding perplexity scores  PP ( X ( T 1 ,A 1 ) ) and PP (  X ( T 2 ,A 1 ) )  obtained after eliminating the outlier pairs. In this setup, a negative t value indicates the presence of a (negative) stereotypical bias. The bias is then statistically significant if the corresponding p -value of the test is within the given confidence interval (in this study set to  = 0 . 05 ). Successful bias mitigation should ideally have no negative effect on the downstream performance of the LM in dialog tasks. We therefore couple the LMB evaluation (3.1) with measures of performance on 1) the original (intrinsic) measurement of in-domain perplexity on Reddit utterances (Zhang et al., 2020), and two dialog tasks: 2) dialog state tracking on MultiWoZ (Budzianowski et al., 2018), and 3) conversational response generation on DSTC-7 (Yoshino et al., 2019). Language Model Perplexity (LMP). Following the original DialoGPT evaluation, we measure the perplexity of the model  before and after we subject it to the bias mitigation methods from 4  on the reference data set consisting of 6 K examples extracted from Reddit by Zhang et al. (2020). 9 Dialog State Tracking (DST). Resorting to one of the central subtasks of task-oriented dialog, we evaluate the models' performances on DST. Here, the goal is to maintain an accurate account of the dialog belief state (i.e., information slots and their values provided by the user) at each turn of the conversation, combining the information from the current user utterance and the conversation history (Henderson et al., 2014; Mrksic et al., 2017). We 9 github.com/microsoft/DialoGPT/blob/ master/data/human.ref.6k.txt evaluate the DST performance on the MultiWoZ 2.0 data set (Budzianowski et al., 2018). 10 As in the original work, DST is cast into a binary prediction task: given the dialog history and the current user utterance, predict for each slot-value combination whether it should be part of the current dialog belief state. As input to DialogGPT, we concatenate the tokens from (i) the previous system output, (ii) the current user utterance, and (iii) the MultiWoZ domain, the slot, and value tokens. We couple the DialoGPT's transformer with a simple feed-forward classifier to which we feed the transformed representation of the last input token. We train the whole model using the binary cross-entropy loss. Conversational Response Generation (CRG). Finally, like the original DialoGPT paper, we evaluate the model  before and after bias mitigation  on the sentence generation task from the Dialog System Technology Challenge 7 (DSTC-7; Yoshino et al., 2019). The models receive (a) a conversational input which includes k most recent preceding turns, and (b) facts  external pieces of texts containing knowledge relevant to the conversation, and are challenged to generate an interesting response that is relevant w.r.t. the dialog history. For simplicity, here we use only the conversational context as input for DialoGPT and ignore the facts. Starting from the transformed representation of the last context token, we then simply fine-tune DialoGPT (transformer encoder plus the LM head) on the train portion of the DSTC-7 data set via causal language modeling, generating the correct response from the data set. The multi-reference test portion of the data set, also created from Reddit, has 5 gold (human) responses for each instance. For evaluating biases and benchmarking bias mitigation effects on REDDITBIAS , we selected the well-known DialoGPT (Zhang et al., 2020) as the conversational LM. Besides being one of the most well-known conversational LMs, it is additionally suitable for evaluation with REDDITBIAS because it was pretrained on Reddit data. We subject DialoGPT to several bias mitigation approaches, which we here adapt in order to make them applicable to conversational LMs. Qian et al. (2019) reduce the gender bias in recurrent LMs by extending the LM loss of the model with an auxiliary term which penalizes differences in probabilities assigned to words from gender pairs, e.g., woman and man . For each of the five bias types (2) and their corresponding bias specifications BE = ( T 1 , T 2 , A 1 , A 2 ) , we manually compile a set of pairs P = { ( t 1 i , t 2 i ) } i  T 1  T 2 for which an unbiased language model should assign equal probability to t 1 i  T 1 and t 2 i  T 2 at the position of any occurrence of either t 1 i or t 2 i . Target terms from both T 1 and T 2 may participate in multiple pairs in P . 11 Let P t  P be the set of pairs in which some target term t (from either T 1 or T 2 ) participates. At every position in which any term t from P occurs, we augment the LM loss with the following debiasing loss: LLMD = 1 | P t | (cid:88) ( t 1 ,t 2)  P i | log  y t 1  y t 2 | , (1) where  y is the predicted probability for a term, with the probability distribution computed only over the reduced vocabulary consisting of terms from P . For positions where any terms from P appears, the overall loss is the weighted sum between the causal LM loss LLM and LLMD : L =  LMLLM +  DLLMD , (2) with the ratio between hyperparameters  LM and  D regulating the trade-off between the language modeling capability and bias mitigation. Inspired by the DebiasNet approach of Lauscher et al. (2020a), applied in the context of debiasing static word embeddings, we devise a debiasing loss that aims to equalize the distance of terms from T 1 and T 2 w.r.t. the stereotypical attribute terms from the attribute set A 1 . For each bias specification, we start from the same set P = { ( t 1 i , t 2 i ) } i  T 1  T 2 of manually created term pairs between the target lists as in the case of LMD. However, this time we focus on occurrences of attribute terms a  A 1 . At every position at which any of the terms from A 1 appears, we augment the LM loss with the 11 E.g., for the bias type Religion #2, we created the following pairs: ( muslim , christian ), ( islamic , christian ), ( islam , christianity ), ( arabs , americans ), ( islamism , christianity ). We list the pairs for all other bias types in the Appendix. Here, a is the transformed vector representation of the token a and t1 and t 2 are vector representations of t 1 and t 2 from the output LM layer (i.e., output embeddings of t 1 and t 2 ), 12 and cos denotes the cosine similarity. ADD forces the output representations of target terms from the dominant group (e.g., christian ) to be equally distant to the representation of a stereotypical attribute for the minoritized group (e.g., dangerous ) as the representations of corresponding target terms denoting the minoritized group (e.g., muslim ). Similar to LMD, for all occurrences of a  A 1 , the final loss is the weighted sum of LLM and LADD , see Eq. (2). Similar to Bordia and Bowman (2019), we next devise a loss based on the idea of hard debiasing from Bolukbasi et al. (2016). We compute this loss in two steps: (1) identification of the bias subspace, and (2) neutralization of the attribute words w.r.t. to the previously identified bias subspace. (1) Bias Subspace Identification. We start from the same set of manually curated target term pairs P as in LMD and ADD. Let t be the output vector of some term t from the LM head. We then obtain partial bias vectors b i for pairs ( t 1 i , t 2 i )  P by computing the differences between t1 i and t2 i : b i = ( t1 i  t2 i ) / 2 . We then stack the partial bias vectors b i to form a matrix C . The bias subspace B then consists of the top k columns of V , obtained via SVD of C (i.e., SVD ( C ) = UV (cid:62) ), with k as the smallest number of singular values that explain at least 50% of the variance of the squared Frobenius norm of the matrix C . (2) Attribute Neutralization. In the second step, we neutralize the contextualized representations of attributes a  A 1 with respect to the bias subspace B computed in the first step. For each occurrence of any a  A 1 , we augment the language modeling loss LLM with the following debiasing loss: LHD = k (cid:88) j =1 | b j (cid:104) a , b j (cid:105)| , (4) 12 For attributes and targets consisting of multiple subword tokens, we average their respective subword vectors. where (cid:104) , (cid:105) denotes the dot product, a is the transformed vector of the input attribute token a , and b j denotes the j -th column of the bias subspace B . The hard debiasing loss forces the transformer network of the language model to produce contextualized representations for stereotypical attributes (e.g., dangerous ) that are orthogonal to k most prominent bias directions. Again, like in LMD and ADD, the total loss for some input token a  A 1 is the weighted sum of the debiasing loss LHD and the language modeling loss LLM . In contrast to the previous three debiasing methods, all of which introduce some type of additional debiasing loss, in CDA (Zhao et al., 2018) we modify the input data on which we fine-tune the DialoGPT via standard causal LM training. The general idea is to break stereotypical associations of the model by duplicating each stereotypical (i.e., biased) instance and then replacing the term denoting the minoritized group with the corresponding term denoting the dominant group. We again start from the manually created set of paired terms P = { ( t 1 i , t 2 i ) } i  T 1  T 2 . For each utterance in the training portion of REDDITBIAS which contains an association between t 1 i  T 1 and a  A 1 (e.g., that Muslim is dangerous ) we create a corresponding counterfactual utterance by replacing t 1 i with its pair t 2 i (e.g., that Christian is dangerous ). We then simply further fine-tune DialoGPT by minimizing the causal LM loss LLM on both the original and counterfactual utterances. In our experiments, we benchmark DialoGPT, a variant of GPT2 (Radford et al., 2019) pretrained on Reddit conversations with the objective to learn to generate responses that are coherent with the contextual prompt. The model is pretrained on a data set containing 147M comment-response pairs spanning the time period from 2005 to 2017. The corpus on which DialoGPT was trained had been preprocessed by removing offensive phrases from a large blacklist. Consequently, DialoGPT is expected to exhibit fewer societal biases than general-purpose language models. We validate this with our evaluation framework based on REDDITBIAS . For each of the five bias types (2) we evaluate  in terms of bias effect and downstream dialog performance (3)  the original DialoGPT and its four debiased variants produced by applying one of the adapted debiasing method (4). Data Splits. For each bias type, we split the set of bias phrases from REDDITBIAS into training, development, and test portions, see Table 3 again. We carry out the debiasing using the training and compute LMB on the test portions of REDDITBIAS . 13 Training and Optimization Details. In all experiments, we use DialoGPT small ( 12 layers, 117 M parameters). For each debiasing run, we train for 2 epochs, and optimize the parameters using Adam (Kingma and Ba, 2015) with the following configu-ration: learning rate = 5  10  5 , weight decay = 0 , beta1 = 0 . 9 , beta2 = 0 . 999 , epsilon = 1  10  8 . In the loss-based debiasing procedures (LMD, ADD, HD) we optimize the hyperparameters on the respective validation portion of REDDITBIAS , searching the following grid: batch size  { 4 , 8 , 16 } , gradient accumulation steps  { 1 , 5 , 8 } ,  LM  { 0 . 001 , 0 . 01 } , and  D  { 10 , 50 , 100 } . We train the downstream models for DST and CRG (3) for a single epoch. We optimize the models using Adam optimizer with the learning rate set to 5  10  5 and epsilon set to 1  10  8 . We limit the input sequences to 128 (subword) tokens. For DST, we train in batches of 48 instances, whereas for CRG, we set the batch size to 80 . Figures 1a and 1b and Tables 4 and 5 summarize our evaluation results. For brevity, we show only F1 scores for DST and Bleu-4 for CRG. 14 13 Note that for CDA, due to the augmentation procedure, we effectively train on two times more utterances. 14 Alternative performance measures, available in the Appendix, show similar trends in results. Stereotypical Bias. As shown in Figure 1a, according to our stereotypical bias measure (LMB), the original DialoGPT model still exhibits significant bias along the dimension of religion, for both Religion #1 ( jews , christians ), and Religion #2 ( muslims , christians ), despite the reported heuristic removal of offensive language from the pretraining data (Zhang et al., 2020). This is most likely due to the more subtle nature of religious stereotypes, which manifest themselves not only in openly offensive text but also in latent co-occurrences of target and attribute terms (e.g., Islam being radical or Jews playing violins ). The bias effect for the Gender dimension is also in the stereotypical direction (i.e., the t-value is negative), but the effect size is insignificant. For Race and Queerness , DialoGPT exhibits insignificant bias effects in the direction opposite from the stereotypical one. We believe that the biases in these two dimensions are most frequently associated with explicit and offensive language, much of which was eliminated in DialoGPT's preprocessing. For the two Religion bias types, in which DialoGPT exhibits significant biases, only two of the four debiasing methods  HD and CDA  are able to remove the stereotypical bias for both bias specifications statistically significantly. LMD and ADD each make the bias insignificant only in one of two cases (LMD for Religion #2 , ADD for Religion #1 ), although they do attenuate the original bias effect for the other specification as well. Interestingly, for the dimensions in which DialoGPT does not exhibit significant stereotypical bias in the first place ( Race , Gender , Orientation ), all four debiasing methods tend to lead to an anti-stereotypical bias effect, i.e., to more strongly (and in a few cases statistically significantly) associated negative stereotypical attributes with the dominant group. For example, criminal gets associated with caucasian , nurse with father or sinful with heterosexual ). This finding stresses the utmost impor-religion1 religion2 race gender orientation 4 2 0 2 4 t _ v a l u e * * * * * * * * DialoGPT LMD ADD HD CDA Downstream Dialog Performance. Encouragingly, none of the four debiasing methods in our study seem to diminish DialoGPT's capabilities in downstream dialog tasks  DST and response generation (see Tables 4 and 5). 15 Interestingly, while LMD drastically increases the perplexity on Reddit utterances (Figure 1b; see LMP in 3) this does not have negative consequences on DST and CRG. To summarize, from the benchmarked debiasing methods, HD and CDA are able to significantly reduce the bias and preserve conversational capabilities; Our results suggest that the dialog performance would remain unaffected even if HD and CDA are to be applied more than once, in order to mitigate multiple bias types. For a comprehensive overview of work on bias in NLP, we refer the reader to (Sun et al., 2019; Blodgett et al., 2020; Shah et al., 2020). Here, we provide (1) a brief overview of bias measures and mitigation methods and their usage in (2) language generation and, specifically, in (3) dialog. (1) Bias in NLP. Resources, measures, and mitigation methods largely target static word embedding models: with their famous analogy man is to computer programmer as woman is to home-maker , Bolukbasi et al. (2016) first drew attention 15 Two exceptions, which requires further investigation are DST performance drops of LMD when debiasing for Race and of ADD when debiasing for Gender . to the issue. Caliskan et al. (2017) presented the Word Embedding Association Test (WEAT), quantifying the bias between two sets of target terms towards two sets of attribute terms. Subsequent work proposed extensions to further embedding models (Liang et al., 2020a,b) and languages (e.g., McCurdy and Serbetci, 2020; Lauscher and Glavas, 2019; Lauscher et al., 2020b; May et al., 2019), analyses of the proposed measures (e.g., Gonen and Goldberg, 2019; Ethayarajh et al., 2019), more comprehensive evaluation frameworks (Lauscher et al., 2020a), new debiasing approaches (Dev and Phillips, 2019; Karve et al., 2019) and task-specific bias measures and resources for tasks like coreference resolution (Zhao et al., 2018), machine translation (Stanovsky et al., 2019) and natural language inference (Dev et al., 2020). In our work, we similarly acknowledge the importance of understanding bias w.r.t. downstream tasks, but focus on dialog systems, for which the landscape of research efforts is surprisingly scarce. (2) Bias in Language Generation. Dialog systems crucially depend on natural language generation (NLG) models. Yeo and Chen (2020) experimented with gender bias in word embeddings for NLG. Sheng et al. (2019) introduce the notion of a regard for a demographic, and compile a data set and devise a bias classification model based on that notion. Webster et al. (2020) proposed Discovery of Correlation (DisCo), a template-based method for gender bias detection which considers an LM's three highest-ranked predictions for a blank text position. Nadeem et al. (2020) introduce StereoSet, a crowdsourced data set for associative contexts at two levels (intra-sentence and inter-sentence) for four bias dimensions. Nangia et al. (2020) present CrowS-Pairs, a data set for measuring bias in masked LMs focusing on nine bias types. However, they don't measure task-oriented model performance, which may degrade as a result of the debiasing procedure (Lauscher et al., 2020a). Qian et al. (2019) reduce gender bias in recurrent LMs with a loss function based on HD (Bolukbasi et al., 2016)  we adapt this method for debiasing conversational LMs (see 4). (3) Bias in Dialog. The landscape of research on bias in dialog systems is scarce: the existing efforts mostly focus on measuring and mitigating gender bias only and do not measure downstream dialog performance of debiased models. Dinan et al. (2020b) focus on multi-dimensional gender bias classification and controlled mitigation. Dinan et al. (2020a) analyze existing dialog data sets for gender bias and extend LIGHT (Urbanek et al., 2019), a resource for grounded dialog, with crowdsourced gender-balanced utterances. Both Lee et al. (2019) and Liu et al. (2020a) add racial bias as a second dimension for bias analysis of dialog models. While Lee et al. (2019) classify whether chatbots agree or disagree with stereotypical statements, Liu et al. (2020a) explore several measures for evaluating bias in dialog systems, including diversity in response generation  this is similar to the work of Liu et al. (2020b) who also include generation quality measures. Overall, these efforts focus only on the two bias dimensions ( gender and race ) and fail to thoroughly analyze the effects of debiasing on performance in dialog tasks such as slot-value extraction, DST, and CRG which are paramount in task-oriented dialog systems. Stereotypical societal biases may lead to the generation of unfair and unethical responses in dialog systems. We presented REDDITBIAS , a comprehensive resource for bias evaluation and debiasing of conversational LMs. Consisting of manually-annotated biased comments from Reddit, REDDITBIAS is the first real-world resource dedicated to multi-dimensional analysis ( gender , race , religion , queerness ) of biases in dialog models. We benchmarked the well-known DialogGPT on REDDITBIAS and analyzed the effects that different debiasing methods (adapted from previous work) have on it. Despite dedicated bias mitigation preprocessing of DialogGPT's pretraining data, it still exhibits prominent religious biases. The benchmarked debiasing methods, however, mostly manage to mitigate those biases, while at the same time retaining the model performance in dialog-oriented downstream tasks (e.g., dialog state tracking). We hope that REDDITBIAS catalyzes research efforts on fair and ethical dialog systems and conversational AI. The work of Anne Lauscher and Goran Glavas has been supported by the Multi2ConvAI Grant (Mehrsprachige und Domanen-ubergreifende Conversational AI) of the Baden-Wurttemberg Ministry of Economy, Labor, and Housing (KI-Innovation). The work of Ivan Vulic has been supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no. 648909) and the ERC PoC Grant MultiConvAI: Enabling Multilingual Conversational AI (no. 957356). Acknowledging the ethical dimension of our work, we like to point the reader to the following limitations and potential implications. (i) Gender is a spectrum and we fully acknowledge the importance of the inclusion of all gender identities , e.g., nonbinary, gender fluid, polygen-der, etc. in language technologies. Note that in our gender bias specification, however, we follow a more classic notion in-line with our focus on the discrepancy between a dominant and a minoritized group. We capture gender identities beyond the binary conception in our LGBTQ bias specification under the notion of queerness . (ii) Similarly important is the intersectional-ity (Crenshaw, 1989) of stereotyping due to the individual composition and interaction of identity chracteristics, e.g., social class and gender (Degaetano-Ortlieb, 2018). Due to its complexity, we do not address the topic in this work. (iii) As we demonstrate in our work, debiasing technologies can, beyond its intended use, be used to increase bias and create biased models. We think that this finding stresses our responsibility to reach out and to raise awareness w.r.t. the impact of language technology among decision makers and users, to establish a broader discourse, and to include ethical aspects in current data science curricula (Bender et al., 2020).
abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain objective abstain objective abstain abstain abstain abstain objective abstain abstain abstain abstain result abstain abstain other method method abstain method method method method method method abstain method method method method abstain method abstain method method method method method method abstain abstain method method abstain method method abstain method method method method abstain method abstain abstain abstain method other abstain other abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method abstain method abstain method abstain method abstain abstain method abstain method method method method method abstain abstain abstain abstain method method abstain method abstain method method abstain abstain method method abstain abstain abstain method abstain method method method abstain abstain abstain abstain abstain method abstain method method abstain method method abstain abstain method abstain abstain method method method method abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method other other other other other method other other other other other other other other abstain other other other other other other other abstain method abstain method abstain abstain abstain other other method abstain method abstain abstain abstain abstain method