Recent developments in Neural Relation Extraction (NRE) have made significant strides towards automated knowledge base construction. While much attention has been dedicated towards improvements in accuracy, there have been no attempts in the literature to evaluate social biases exhibited in NRE systems. In this paper, we create WikiGenderBias, a distantly supervised dataset composed of over 45,000 sentences including a 10% human annotated test set for the purpose of analyzing gender bias in relation extraction systems. We find that when extracting spouse and hypernym (i.e., occupation) relations, an NRE system performs differently when the gender of the target entity is different. However, such disparity does not appear when extracting relations such as birth date or birth place. We also analyze two existing bias mitigation techniques, word embedding debiasing and data augmentation. Unfortunately, due to NRE models relying heavily on surface level cues, we find that existing bias mitigation approaches have a negative effect on NRE. Our analysis lays groundwork for future quantifying and mitigating bias in relation extraction. With the wealth of information being posted online daily, relation extraction has become increasingly important. Relation extraction aims specifically to extract relations from raw sentences and represent them as succinct relation tuples of the form (head, relation, tail) e.g., (Barack Obama, spouse, Michelle Obama) . The concise representations provided by relation extraction models have been used to extend Knowledge Bases (KBs) (Riedel et al., 2013; Subasic et al., 2019; Trisedya et al., 2019). These KBs are then used heavily in NLP systems, such as question answering systems (Bordes et al., 2014; Yin et al., 2016; Cui et al., 2019). In recent years, much focus in the Neural Relation Extraction (NRE) community has been centered on improvements in model precision and the reduction of noise (Lin et al., 2016; Liu et al., 2017; Wu et al., 2017; Feng et al., 2018; Vashishth et al., 2018; Qin et al., 2018). Yet, little attention has been devoted towards the fairness of such systems. We take the first step at understanding and evaluating gender bias in NRE systems by measuring the differences in model performance when extracting relations from sentences written about females versus sentences written about males. If a NRE model predicts a relation such occupation with higher recall on male entities, this could lead to the resulted knowledge bases having more occupation information for males than for females (see the illustration in Figure 1). Eventually, the gender bias in knowledge bases may affect downstream predictions, causing undesired allocative harms (Craw-ford, 2017) and reinforcing gender-stereotypical beliefs in society. In this paper, we present an evaluation framework to analyze social bias in NRE models. Specifically, we evaluate gender bias in English language predictions of a collection of popularly used and open source NRE models 1 (Lin et al., 2016; Wu et al., 2017; Liu et al., 2017; Feng et al., 2018). We evaluate on two fronts: (1) examining gender bias 1 https://github.com/thunlp/OpenNRE/ Figure 1: An illustration of gender bias in relation extraction and how it affects a downstream application. exhibited in a model that is trained on a relation extraction dataset; and (2) examining if the existing bias mitigation techniques (Bolukbasi et al., 2016; Zhao et al., 2018; Lu et al., 2018) can be applied to reduce the bias in an NRE system while maintaining its performance. Carrying out such an evaluation is difficult with existing NRE datasets, such as the NYT dataset (Sandhaus, 2018), because there is no reliable way to obtain gender information about the entities mentioned in input sentences. Therefore, we create a new dataset, WikiGenderBias, specifically aimed at evaluating gender bias for NRE. WikiGenderBias is a distantly supervised dataset extracted using Wikipedia and DBPedia. It contains 45,000 sentences, each of which describe either a male or female entity with one of four relations: spouse, hypernym (i.e., occupation), birthDate, and birthPlace . We posit that a biased NRE system leverages gender information as a proxy when extracting knowledge tuples with spouse and hypernym relations. However, gender of the entity does not affect the extraction of relations such as birthDate and birthPlace, as they are not intuitively related to gender. Experiment results confirm our conjecture. Our contributions are as such:  We create WikiGenderBias, a new dataset for evaluating gender bias in NRE systems. We present an evaluation framework to demonstrate that gender bias is exhibited in NRE model outputs. We test several existing bias mitigation approaches to reducing gender bias in NRE system. Our analysis sheds light for designing future mitigating techniques. Gender Bias Measurement. Existing studies have revealed gender bias in various NLP tasks (Zhao et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; Dixon et al., 2018; Lu et al., 2018; Kiritchenko and Mohammad, 2018; Romanov et al., 2019; Sheng et al., 2019; Sun et al., 2019). People have proposed different metrics to evaluate gender bias, for example, by using the performance difference of the model on male and female datapoints for bias evaluation (Lu et al., 2018; Kiritchenko and Mohammad, 2018). Other metrics have been proposed to evaluate fairness of predictors and allocative bias (Dwork et al., 2012; Hardt et al., 2016), such as Equality of Opportunity. In this work, we use both of these metrics to evaluate NRE models. Mitigation Methods. After discovering gender bias existing, prior work has developed various methods to mitigate that bias (Escude Font and Costa-juss`a, 2019; Bordia and Bowman, 2019). Those mitigation methods can be applied in different levels of a model, including in the training phase, in the embedding layer, or in the inference procedure. In this paper, we test three existing debiasing approaches, namely data augmentation (Zhao et al., 2018; Lu et al., 2018), and word embedding debiasing technique (Hard Debiasing (Bolukbasi et al., 2016)) for mitigating bias in NRE models. Neural Relation Extraction. Relation extraction is a task in NLP with a long history that typically seeks to extract structured tuples ( e 1 , r, e 2 ) from texts (Bach and Badaskar, 2007). Early on, learning algorithms for relation extraction models were typically categorized as supervised, including feature-based methods (Kambhatla, 2004; Zhou et al., 2005; Zhao and Grishman, 2005) and kernel-based methods (Lodhi et al., 2002; Zelenko et al., 2003), or semi-supervised (Brin, 1998; Agichtein and Gravano, 2000; Etzioni et al., 2005; Pantel and Pennacchiotti, 2006), or purely unsupervised (Et-zioni et al., 2008). Supervised approaches suffer from the need for large amounts of labelled data, which is sometimes not feasible, and generalizes poorly to open domain relation extraction, since labeled data is required for every entity-relation type (Bach and Badaskar, 2007; Mintz et al., 2009). Many semi-supervised approaches rely on pattern-matching, which is not robust, and many are unable to extract intra-sentence relations (Bach and Badaskar, 2007). When data annotation is insuf-ficient or hard to obtain and semi-supervised approaches are insufficient, the distant supervision assumption is used to collect data to train supervised models (Mintz et al., 2009). Given a relation ( e 1 , r, e 2 ) in a knowledge base (KB), distant supervision assumes any sentence that contains both e 1 and e 2 expresses r (Mintz et al., 2009). Great efforts have been made to improve NRE models by mitigating the effects of noise in the training data introduced by Distant Supervision (Hoffmann et al., 2011; Surdeanu et al., 2012; Lin et al., 2016; Liu et al., 2017; Feng et al., 2018; Qin et al., 2018). However, to our knowledge, there are no studies on bias or ethics in NRE, which is filled by this work. We define gender bias in NRE as a difference in model performance when predicting on sentences from male versus female articles. Thus, we need articles written about entities for which we can identify the gender information. However, to obtain gender information for existing annotated datasets could be costly or impossible. Thus, we elected to create WikiGenderBias with this gender information to be able to detect scenarios like that in Figure 1. The data statistics of WikiGenderBias are given in Table 1. Wikipedia is associated with a knowledge base, DBPedia, that contains relation information for entities with articles on Wikipedia (Mendes et al., 2012). Many of these entities have gender information and their corresponding articles are readily available. Therefore, we create our dataset based on sentences extracted from Wikipedia. To generate WikiGenderBias, we use a variant of the distant supervision assumption: for a given relation between two entities, if one sentence from an article written about one entity also mentions the other entity, then we assume that such sentence expresses the relation. For instance, if we know (Barack, spouse, Michelle) is a relation tuple and we find the sentence He and Michelle were married in Barack's Wikipedia article, then we assume that sentence expresses the (Barack, spouse, Michelle) relation. This assumption is similar to that made by Mintz et al. (2009) and allows us to scalably create the dataset. WikiGenderBias considers four relations that stored in DBPedia: spouse , hypernym , birthDate , Relation Head Entity Tail Entity Sentence Birthdate Robert M. Kimmitt December 19, 1947 Robert M. Kimmitt ( born December 19 , 1947 ) was United States Deputy Secretary of the Treasury under President George W. Bush . and birthPlace . Note that the hypernym relation on DBPedia is similar to occupation, with entities having hypernym labels such as Politican. We also generate negative examples by obtaining datapoints for three unrelated relations: parents , deathDate , and almaMater . We label them as NA (not a rela-tion). As each sentence only labelled with one relation based on our distant supervision assumption, WikiGenderBias is a 5-class classification relation extraction task. Figure 2 lists the label distribution. We hypothesize that a biased relation extraction model might use gender as a proxy to influence predictions for spouse and hypernym relations, since words pertaining to marriage are more often mentioned in Wikipedia articles about female entities and words pertaining to hypernym (which is similar to occupation) are more often mentioned in Wikipedia articles about male entities (Wagner et al., 2015; Graells-Garrido et al., 2015). On the other hand, we posit that birthDate and birthPlace would operate like control groups and believe gender would correlate with neither relation. To simplify the analysis, we only consider the head entities that associated with at least one of the four targeted relations. We set up our experiment such that head entities are not repeated across the train, dev, and test sets so that the model will see only new head entities at the test time. Since we obtain the distantly supervised sentences for a relation from the head entity's article, this guarantees the model will not reuse sentences from an article. However, it is possible that the head entity will appear as a tail entity in other relations because an entity could appear in multiple articles. The data splits are given in Table 1. there are more male instances than female instances in WikiGenderBias as well. To remove the effect of dataset bias in our analysis, we also create a gender-equalized version of the training and development sets by down-sampling male instances. We discuss the creation of gender-equalized test set below. We equalize the male and female instances in the test set. In this way, a model cannot achieve high performance by performing well only on the dominant class. Furthermore, since some data instances that are collected using distant supervision are noisy, we annotated the correctness of the test instances using Amazon Mechanical Turk annotations to perform a fair comparison. Specifically, we asked workers to determine whether or not a given sentence expressed a given relation. If the majority answer was no, then we labeled that sentence as expressing no relation (we denote them as NA). Each sentence was annotated by three workers. Each worker was paid 15 cents per annotation. We only accepted workers from England, the US or Australia and with HIIT Approval Rate greater than 95% and Number of HIITs greater than 100 . We found the pairwise inter-annotator agreement as measured by Fleiss' Kappa (Fleiss, 1971)  is 0.44, which is consistent across both genders and signals moderate agreement. We note that our  value is affected by asking workers to make binary classifications, which limits the degree of agreement that is attainable above chance. We also found the pairwise inter-annotator agreement to be 84%. We build on the work of Graells-Garrido et al. (2015), who discovered that female entities are more likely to have spouse information in the In-foboxes on their Wikipedia page than male entities. Figure 2 demonstrates a further discrepancy: amongst articles we sampled, proportionally, the spouse relation is mentioned more often relative to hypernym, birthPlace, and birthDate in female articles than in male articles. Additionally, we show that amongst female and male articles we sampled, hypernyms are mentioned more often in male than female articles relative to spouse, birthPlace, and birthDate (see Section 2). This observation aligns with the literature, arguing that authors do not write about the two genders equally (Wagner et al., 2015; Graells-Garrido et al., 2015). We evaluate OpenNRE (Han et al., 2019), a popular open-source NRE system. OpenNRE implements the approach from (Lin et al., 2016). To convert sentences into vectors, researchers propose convolutional neural networks as well as the pieceweise convoultional neural networks (PCNN) which retain more structural information between entities (Zeng et al., 2015). In this work, we use a PCNN with Selective Attention for the experiments. We train every encoder-selector combination on the training set of WikiGenderBias and its gender-equalized version. We input Word2Vec (Mikolov et al., 2013) word embeddings trained on WikiGenderBias to the models 2 . The goal of a successful relation extraction model is to maximize F1 score while minimizing the model performance gender gap (or disparity score). However, when comparing different systems, it is hard to decide what is the right balance between these two objectives. On one end, a model which has zero gender gap but has only 10% accuracy for both male and female test instances has almost no practical value. Other methods that have high accuracy or F1 score may do so at the cost of a wide gender gap. Although our test set for WikiGenderBias is gender-equalized, one can imagine that improving performance on a test set that is heavily skewed towards males can be done by focusing on male test instances while largely ignoring female ones. Therefore, it is important to strike a balance between model performance and inter-group parity. To measure model performance, we use Macro-average F1 score. To measure inter-group parity, we use the pairwise difference in F1 scores averaged over all the groups for predictions on a given relation i . We describe the average difference over all relations as Disparity Score (DS): DS = 1 n n (cid:88) i =1 1 x x (cid:88) j =1 x (cid:88) k = j +1 (cid:12)(cid:12)(cid:12)(cid:12) F 1 ik  F 1 ij (cid:12)(cid:12)(cid:12)(cid:12) , where n denotes the number of relations (e.g. { birthDate, birthPlace, spouse, hypernym } ). x denotes the number of groups (e.g. { male, female } ). F 1 rk is the F 1 score for the model when predicting datapoints with true label relation r that belong to group k . (So, for instance, F 1 spouse,male is the F 1 score on sentences that express the spouse relation from male articles.) The Disparity Score measures the F1 score gap between predictions on male and female data points. Bringing these two metrics together, we propose the Performance Parity Score (PPS). 2 We performed Grid Search to determine the optimal hy-perparameters. We set epochs = 60 , learning rate  = 0 . 5 , early stopping with patience of 10, batch size = 160 , and sliding window size = 3 (for CNN and PCNN). These hyperpa-rameters are similar to the default settings found in the OpenNRE repository tensorflow branch, which uses epochs = 60 , learning rate  = 0 . 5 , and early stopping with patience of 20. F1 score subtracted by the model performance gender gap, which we defined as the Disparity Score, per relation. We place equal importance on the F1 score and Disparity Score by giving each score an implicit weight of 1 . In our formula for PPS above, we also divide the final result by the number of relations n . This keeps the range of PPS within (  1 , 1] , although PPS will generally fall between [0 , 1] because it is highly unlikely that the Disparity Score will be greater than the overall F1 score. PPS seeks to incentivize a combination of both model performance and inter-group parity for the task of relation extraction: PPS = 1 n n (cid:88) i =1 (cid:18) F 1 i  1 x x (cid:88) j =1 x (cid:88) k = j +1 (cid:12)(cid:12)(cid:12)(cid:12) F 1 ik  F 1 ij (cid:12)(cid:12)(cid:12)(cid:12)(cid:19) = 1 n n (cid:88) i =1 F 1 i  1 n n (cid:88) i =1 1 x x (cid:88) j =1 x (cid:88) k = j +1 (cid:12)(cid:12)(cid:12)(cid:12) F 1 ik  F 1 ij (cid:12)(cid:12)(cid:12)(cid:12) = Macro F 1 score  Disparity Score . Similar to the parity term in PPS, gender bias can be measured as the difference in a performance metric for a model when evaluated on male and female datapoints (De-Arteaga et al., 2019). We define male (female) datapoints to be relations for which the head entity is male (female), which means the distantly supervised sentence is taken from a male (female) article. Prior work has used area under the precision-recall curve and F1 score to measure NRE model performance (Gupta et al., 2019; Han et al., 2019; Kuang et al., 2019). We use Macro-F1 score as our performance metric. We denote the F1 gender difference as F 1 Gap , which is used to calculate the disparity score. A larger disparity score indicates higher bias in predictions. Equality of Opportunity (EoO) was originally proposed to measure and address allocative biases (Hardt et al., 2016). Consequently, we examine this metric in the context of relation extraction to better understand how allocative biases can begin to emerge at this stage. Equality of Opportunity (EoO) is defined in terms of the joint distribution of ( X, A, Y ) , where X is the input, A is a protected attribute that should not influence the prediction, and Y is the true label (Hardt et al., 2016). A predictor satisfies Equality of Opportunity if and only if: P (  Y = 1 | A = male, Y = 1) and P (  Y = 1 | A = female, Y = 1) . In our case A = { male, female } , because gender is our protected attribute and we assume it to be binary. We evaluate EoO on a per-relation, one-versus-rest basis. Thus, when calculating EoO for spouse, Y = 1 indicates the true label is spouse and  Y = 1 indicates a prediction of spouse. We do this for each relation. Note that this is equivalent to measuring per-relation recall for each gender. As shown in Figure 3, the NRE system performs better when predicting the spouse relation on sentences from articles about male entities than from articles on female entities (see Figure 3, right). Further, there is a large recall gap (see EoO column, row 1, in Table 3). Notably, the gender difference in performance is much smaller on birthDate, birthPlace, and hypernym relations, although the gender difference is non-zero for birthPlace and hyerpym. This is interesting given that a higher percentage of female instances in WikiGenderBias are spouse relations than male (see Figure 2). We encourage future work to explore whether the writing style differences between male and female spouse in-Spouse Birth Date Birth Place Hypernym Total F 1 Gap EoO F 1 Gap EoO F 1 Gap EoO F 1 Gap EoO F1 Score Disparity Score PPS PCNN,ATT .041 .058 .004 .000 -.003 -.017 .015 .009 .886 .016 .870 CNN,ATT .034 .043 -.003 .001 .014 .004 .028 .014 .882 .020 .862 RNN,ATT .032 .043 .015 .019 .005 -.011 -.006 -.006 .889 .014 .875 BIRNN,ATT .039 .061 .013 .021 -.016 -.033 -.013 -.026 .884 .020 .864 PCNN,AVE .034 .044 .005 .010 -.001 -.011 .005 -.005 .903 .011 .892 CNN,AVE .027 .028 .013 .029 .007 .009 .002 -.028 .895 .012 .883 RNN,AVE .039 .036 .004 .021 .016 .020 .006 -.012 .912 .016 .895 BIRNN,AVE .024 .018 .001 .015 .009 .018 -.005 -.022 .913 .010 .903 Table 3: Results from running combinations of encoders and selectors of the OpenNRE model for the male and female genders of each relation. In addition, we explore different types of sentence encoder and sentence-level attention used in the creation of the bag representation for each entity pair and examined how these models performed on our dataset. Notably, the bias in spouse relation persists across OpenNRE architectures (see Table 3). It seems models using average attention, which merely averages all the sentence vectors in the bag to create a representation of the entire bag, allows for better aggregate performance on WikiGenderBias. However, the effect on the Disparity Score (and therefore the bias exhibited in the predictions) seems negligible. We note that these results do not necessarily indicate that the model itself contains biases given that males and females are written about differently on Wikipedia. These results do, however, demonstrate that we must be cautious when deploying NRE models, especially those trained on Wikipedia data, since they can propagate biases latent in their training data to the knowledge bases they help create. Equalizing the Gender Distribution Sometimes, the true distribution contains an imbalance in gendered data. For instance, perhaps the training set contains more instances from male articles than female. To mitigate this, one can simply downsam-ple the male instances until the male and female instances are approximately equal, then train on this modified, equalized distribution. Data Augmentation. The contexts in which males and females are written about can differ; for instance, on Wikipedia women are more often written about with words related to sexuality than men (Graells-Garrido et al., 2015). Data augmentation mitigates these contextual biases by replacing mas-culine words in a sentence with their corresponding feminine words and vice versa for all sentences in a corpus, and then training on the union of the original and augmented corpora 3 (Zhao et al., 2018; Lu et al., 2018; Dixon et al., 2018; Maudslay et al., 2019; Zhao et al., 2019). Word Embedding Debiasing Word embeddings can encode gender biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2018) and this can affect bias in downstream predictions for models using the embeddings (Zhao et al., 2018; Font and Costa-Jussa, 2019). In this work, we apply the Hard-Debiasing technique (Bolukbasi et al., 2016). We applied Hard-Debiasing to Word2Vec embeddings (Mikolov et al., 2013), which we trained on the sentences in WikiGenderBias. When used in conjunction with data augmentation, the embeddings are re-trained on the union of the two corpora. Below, we give metrics used for measuring model performance and bias in our experiments. 3 We use the following list to perform data augmentation: https://github.com/uclanlp/corefBias/ blob/master/WinoBias/wino/generalized_swaps.txt Figure 4: Bias in relation extraction model on each relation as measured by male  female F1 score gender gap (used to calculate disparity score) for the default training set without modifications (left) and equalized training set (right). We note that by downsampling the training instances to equalize the number of male and female datapoints, the difference in performance on male versus female sentences decreases to almost 0 for every relation aside from hypernym (see Figure 4, right). Additionally, the drop in aggregate is performance is relatively small (see Macro F1, Table 4). Given that we down-sampled male instances to create this equalized dataset, training on the equalized data was also more efficient. We also examined the effect of various debiasing techniques. Table 4 shows the results. Unfortunately, most of these techniques cause a significant performance drop and none of them is effective in reducing the performance gap between genders. Interestingly, debiasing embeddings increased aggregate performance by achieving slightly better F1 performance. As none of these mitigation approaches is effective, their combinations are not effective as well. They either lowering Macro F1 or raising Disparity Score or both. We further examine the performance of various bias mitigation techniques evaluated in each relation in Figure 4. NRE relies heavily on surface-level cues such as context, the entities, and their positions. Data augmentation might potentially introduce artifacts and biases, causing the NRE system captures unwanted patterns and spurious statistics between contexts. In our study, we create and publicly release WikiGenderBias: the first dataset aimed at evaluating bias in NRE models. We train NRE models on the WikiGenderBias dataset and test them on gender-separated test sets. We find a difference in F1 scores for the spouse relation between predictions on male sentences and female for the model's predictions. We also examine existing bias mitigation techniques and find that naive data augmentation causes a significant performance drop. It is an open and difficult research question to build unbiased neural relation extractors. One possibility is that some bias mitigation methods that add noise to the dataset encourage neural relation extraction models to learn spurious correlations and unwanted biases. We encourage future work to dive deeper into this problem. While these findings will help future work avoid gender biases, this study is preliminary. We only consider binary gender, but future work should consider non-binary genders. Additionally, future work should further probe the source of gender bias in the model's predictions, perhaps by visualizing attention or looking more closely at the model's outputs. We thank anonymous reviewers for their helpful feedback. This material is based upon work supported in part by the National Science Foundation under IIS Grant 1927554 and Grant 1821415: Scaling the Early Research Scholars Program.
abstain abstain objective result abstain method result method abstain abstain abstain abstain abstain abstain objective abstain abstain method method other abstain abstain objective abstain abstain objective abstain result objective objective method method other other other other method other other other method other other other other other other other other abstain method method abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain method method abstain method other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain objective method result result abstain abstain method abstain method abstain other other