Online misogyny, a category of online abusive language, has serious and harmful social consequences. Automatic detection of misogynistic language online, while imperative, poses complicated challenges to both data gathering, data annotation, and bias mitigation, as this type of data is linguistically complex and diverse. This paper makes three contributions in this area: Firstly, we describe the detailed design of our iterative annotation process and codebook. Secondly, we present a comprehensive taxonomy of labels for annotating misogyny in natural written language, and finally, we introduce a high-quality dataset of annotated posts sampled from social media posts. Abusive language is a phenomenon with serious consequences for its victims, and misogyny is no exception. According to a 2017 report from Amnesty International, 23% of women from eight different countries have experienced online abuse or harassment at least once, and 41% of these said that on at least one occasion, these online experiences made them feel that their physical safety was threatened (Amnesty International, 2017). Automatic detection of abusive language can help identify and report harmful accounts and acts, and allows counter narratives (Chung et al., 2019; Garland et al., 2020; Ziems et al., 2020). Due to the volume of online text and the mental impact on humans who are employed to moderate online abusive language moderators of abusive online content have been shown to develop serious PTSD and depressive symptoms (Casey Newton, 2020) it is urgent to develop systems to automate the detection and moderation of online abusive language. Automatic detection, however, presents significant challenges (Vidgen et al., 2019). Abusive language is linguistically diverse (Vid-gen and Derczynski, 2020), both explicitly, in the form of swear words or profanities; implicitly, in the form of sarcasm or humor (Waseem et al., 2017); and subtly, in the form of attitudes and opinions. Recognizing distinctions between variants of misogyny is challenging for humans, let alone computers. Systems for automatic detection are usually created using labeled training data (Kiritchenko et al., 2020), hence, their performance depends on the quality and representativity of the available datasets and their labels. We currently lack transparent methods for how to create diverse datasets. When abusive language is annotated, classes are often created based on each unique dataset (a purely inductive approach), rather than taking advantage of general, established terminology from, for instance, social science or psychology (a deductive approach, building on existing research). This makes classification scores difficult to compare and apply across diverse training datasets. This paper investigates the research question: How might we design a comprehensive annotation process which results in high quality data for automatically detecting misogyny? We make three novel contributions: 1. Methodology: We describe our iterative approach to the annotation process in a transparent way which allows for a higher degree of comparability with similar research. 2. Model: We present a taxonomy and annotation codebook grounded in previous research on automatic detection of misogyny as well as social science terminology. 3. Dataset: We present a new, annotated corpus of Danish social media posts, Bajer, 1 annotated for misogyny, including analysis of class balance, word frequencies, Inter-Annotator Agreement (IAA), annotation errors, and classification baseline. 1 https://github.com/phze22/ Online-Misogyny-in-Danish-Bajer Since research has indicated that misogyny presents differently across languages, and, likely, cultures (Anzovino et al., 2018), an additional contribution of this work is that it presents a dataset of misogyny in Danish , a North Germanic language, spoken by only six million people, and indeed the first work of its kind in any Scandina-vian/Nordic culture to our knowledge. In Denmark an increasing proportion of people refrain from online discourse due to the harsh tone, with 68% of social media users self-excluding in 2021 (Anal-yse & Tal, 2021; Andersen and Langberg, 2021), making this study contextually relevant. Further, the lack of language resources available for Danish (Kirkedal et al., 2019) coupled with its lexical complexity (Bleses et al., 2008) make it an intricate research objective for natural language processing. Abusive language is as ancient a phenomenon as written language itself. Written profanities and insults about others are found as old as graffiti on ruins from the Roman empire (Wallace, 2005). Automatic processing of abusive text is far more recent, early work including e.g. Davidson et al. (2017) and Waseem et al. (2017). Research in this field has produced both data, taxonomies, and methods for detecting and defining abuse, but there exists no objective framing for what constitutes abuse and what does not. In this work, we focus on a specific category of online abuse, namely misogyny . Misogyny can be categorised as a subbranch of hate speech and is described as hateful content targeting women (Waseem, 2016). The degree of toxicity depends on complicated subjective measures, for instance, the receiver's perception of the dialect of the speaker (Sap et al., 2019). Annotating misogyny typically requires more than a binary present/absent label. Chiril et al. (2020), for instance, use three categories to classify misogyny in French: direct sexist content (directly addressed to a woman or a group of women), descriptive sexist content (describing a woman or women in general) or reporting sexist content (a report of a sexism experience or a denunciation of a sexist behaviour). This categorization does not, however, specify the type of misogyny. from the work of Waseem and Hovy (2016). While harsh sexism (hateful or negative views of women) is the more recognized type of sexism, benevolent sexism (a subjectively positive view towards men or women), often exemplified as a compliment using a positive stereotypical picture, is still discriminating (Glick and Fiske, 1996). Other cat-egorisations of harassment towards women have distinguished between physical, sexual and indirect occurrences (Sharifirad and Jacovi, 2019). Anzovino et al. (2018) classify misogyny more segregated in five subcategories: Discredit , Harassment & Threats of Violence , Derailing , Stereotype & Objectification , and Dominance . They also distinguish between if the abuse is active or passive towards the target. These labels appear to apply well to other languages, and quantitative representation of labels differ by language. For example, Spanish shows a stronger presence of Dominance , Italian of Stereotype & Objectification , and English of Discredit . As we see variance across languages, building terminology for labeling misogyny correctly is therefore a key challenge in being able to detect it automatically. Parikh et al. (2019) take a multi-label approach to categorizing posts from the Everyday Sexism Project, where as many as 23 different categories are not mutually exclusive. The types of sexism identified in their dataset include body shaming , gaslighting , and mansplain-ing . While the categories of this work are extremely detailed and socially useful, several studies have demonstrated the challenge for human annotators to use labels that are intuitively unclear (Chatzakou et al., 2017; Vidgen et al., 2019) or closely related to each other (Founta et al., 2018). Guest et al. (2021) suggest a novel taxonomy for misogyny labeling applied to a corpus of primarily English Reddit posts. Based on previous research, including Anzovino et al. (2018), they present the following four overarching categories of misogyny: (i) Misogynistic Pejoratives, (ii) descriptions of Misogynistic Treatment, (iii) acts of Misogynistic Derogation and (iv) Gendered Personal attacks against women. The current work combines previous categorizations on misogyny into a taxonomy which is useful for annotation of misogyny in all languages, while being transparent about the construction of this taxonomy. Our work builds on the previous work presented in this section, continuous discussions among the annotators, and the addition of social science terminology to create a single-label taxonomy of misogyny as identified in Danish social media posts across various platforms. The creation of quality datasets involves a chain of methodological decisions. In this section, we will present the rationale of creating our dataset under three headlines: Dataset, Annotation process, and Mitigating biases. Bender and Friedman (2018) present a set of data statements for NLP which help alleviate issues related to exclusion and bias in language technology, lead[ing] to better precision in claims about how natural language processing research can generalize and thus better engineering results. Data statements are a characterization of a dataset which provides context to others to understand how experimental results might generalize and what biases might be reflected in systems built on the software. We present our data statements for the dataset creation in the following: Curation rationale: Random sampling of text often results in scarcity of examples of specifically misogynistic content (e.g. (Wulczyn et al., 2017; Founta et al., 2018)). Therefore, we used the common alternative of collecting data by using pre-defined keywords with a potentially high search hit (e.g. Waseem and Hovy (2016)), and identifying relevant user-profiles (e.g. (Anzovino et al., 2018)) and related topics (e.g. (Kumar et al., 2018)). We searched for keyword (specific slurs, hash-tags), that are known to occur in sexist posts. These were defined by previous work, a slur list from Reddit, and from interviews and surveys of online misogyny among women. We also searched for broader terms like sex or women, which do not appear exclusively in a misogynistic context, for example in the topic search, where we gathered relevant posts and their comments from the social media pages of public media. A complete list of keywords can be found in the appendix. Social media provides a potentially biased, but broad snapshot of online human discourse, with plenty of language and behaviours represented. Following best practice guidelines (Vidgen and Derczynski, 2020), we sampled from a language for which there are no existing annotations of the target phenomenon: Danish. Different social media platforms attract different user groups and can exhibit domain-specific language (Karan and Snajder, 2018). Rather than choosing one platform (existing misogyny datasets are primarily based on Twitter and Reddit (Guest et al., 2021)), we sampled from multiple platforms: Statista (2020) shows that the platform where most Danish users are present is Facebook, followed by Twitter, YouTube, Instagram and lastly, Reddit. The dataset was sampled from Twitter, Facebook and Reddit posts as plain text. Text characteristics: Danish colloquial web speech. Posts, comments, retweets: max. length 512, average length: 161 characters. Annotator demographics: We recruited annotators aiming specifically for diversity in gender, age, occupation/ background (linguistic and ethnographic knowledge), region (spoken dialects) as well as an additional facilitator with a background in ethnography to lead initial discussions (see Table 1). Annotators were appointed as full-time employees with full standard benefits. In annotating our dataset, we built on the MATTER framework (Pustejovsky and Stubbs, 2012) and use the variation presented by Finlayson and Erjavec (2017) (the MALER framework), where the Train We created a set of guidelines for the annotators. The annotators were first asked to read the guidelines and individually annotate about 150 different posts, after which there was a shared discussion. After this pilot round, the volume of samples per annotator was increased and every sample labeled by 2-3 annotators. When instances were flagged' or annotators disagreed on them, they were discussed during weekly meetings, and misunderstandings were resolved together with the external facilitator. After round three, when reaching 7k annotated posts (Figure 2), we continued with independent annotations maintaining a 15% instance overlap between randomly picked annotator pairs. Management of annotator disagreement is an important part of the process design. Disagreements can be solved by majority voting (Davidson et al., 2017; Wiegand et al., 2019), labeled as abuse if at least one annotator has labeled it (Golbeck et al., 2017) or by a third objective instance (Gao and Huang, 2017). Most datasets use crowdsourcing platforms or a few academic experts for annotation (Vidgen and Derczynski, 2020). Inter-annotator-agreement (IAA) and classification performance are established as two grounded evaluation measurements for annotation quality (Vidgen and Derczynski, 2020). Comparing the performance of amateur annotators (while providing guidelines) with expert annotators for sexism and racism annotation, Waseem (2016) show that the quality of amateur annotators is competitive with expert annotations when several amateurs agree. Facing the trade-off between training annotators intensely and the number of involved annotators, we continued with the trained annotators and group discussions/ individual revisions for flagged content and disagreements (Section 5.4). Prior work demonstrates that biases in datasets can occur through the training and selection of annotators or selection of posts to annotate (Geva et al., 2019; Wiegand et al., 2019; Sap et al., 2019; Al Kuwatly et al., 2020; Ousidhoum et al., 2020). Selection biases: Selection biases for abusive language can be seen in the sampling of text, for instance when using keyword search (Wiegand et al., 2019), topic dependency (Ousidhoum et al., 2020), users (Wiegand et al., 2019), domain (Wiegand et al., 2019), time (Florio et al., 2020) and lack of linguistic variety (Vidgen and Derczynski, 2020). Label biases: Label biases can be caused by, for instance, non-representative annotator selection, lack in training/domain expertise, preconceived notions, or pre-held stereotypes. These biases are treated in relation to abusive language datasets by several sources, e.g. general sampling and annotators biases (Waseem, 2016; Al Kuwatly et al., 2020), biases towards minority identity mentions based for example on gender or race (Davidson et al., 2017; Dixon et al., 2018; Park et al., 2018; Davidson et al., 2019), and political annotator biases (Wich et al., 2020). Other qualitative biases comprise, for instance, demographic bias, over-generalization, topic exposure as social biases (Hovy and Spruit, 2016). Systematic measurement of biases in datasets remains an open research problem. Friedman and Nissenbaum (1996) discuss freedom from biases as an ideal for good computer systems, and state that methods applied during data creation influence the quality of the resulting dataset quality with which systems are later trained. Shah et al. (2020) showed that half of biases are caused by the methodology design, and presented a first approach of classifying a broad range of predictive biases under one umbrella in NLP. We applied several measures to mitigate biases occurring through the annotation design and execution: First, we selected labels grounded in existing, peer-reviewed research from more than one field. Second, we aimed for diversity in annotator profiles in terms of age, gender, dialect, and background. Third, we recruited a facilitator with a background in ethnographic studies and provided intense annotator training. Fourth, we engaged in weekly group discussions, iteratively improving the codebook and integrating edge cases. Fifth, the selection of platforms from which we sampled data is based on local user representation in Denmark, rather than convenience. Sixth, diverse sampling methods for data collection reduced selection biases. Good language taxonomies systematically bring together definitions and describe general principles of each definition. The purpose is categorizing reference lang. and mapping entities in a way that demonstrates their natural relationship, e.g. Schmidt and Wiegand (2017); Anzovino et al. (2018); Zampieri et al. (2019); Banko et al. (2020). Their application is especially clear in shared tasks, as for multilingual sexism detection against women, SemEval 2019 (Basile et al., 2019). On one hand, it should be an aim of a taxonomy that it is easily understandable and applicable for annotators from various background and with different expertise levels. On the other hand, a taxonomy is only useful if it is also correct and comprehensive , i.e. a good representation of the world. Therefore, we have aimed to integrate definitions from several sources of previous research (deductive approach) as well as categories resulting from discussions of the concrete data (inductive approach). Our taxonomy for misogyny is the product of (a) existing research in online abusive language and misogyny (specifically the work in Table 2), (b) a review of misogyny in the context of online platforms and online platforms in a Danish context (c) iterative adjustments during the process including discussions between the authors and annotators. The labeling scheme (Figure 1) is the main structure for guidelines for the annotators, while a codebook ensured common understanding of the label descriptions. The codebook provided the annotators with definitions from the combined taxonomies. The descriptions were adjusted to distinguish edge-cases during the weekly discussion rounds. The taxonomy has four levels: (1) Abusive (abusive/not abusive), (2) Target (indi-vidual/group/others/untargeted), (3) Group type (racism/misogyny/others), (4) Misogyny type (harassment/discredit/stereotype & objectifica-tion/dominance/neosexism/benevolent). To demonstrate the relationship of misogyny to other instances of abusive language, our taxonomy embeds misogyny as a subcategory of abusive language. Misogyny is distinguished from, for instance, personal attacks, which is closer to the abusive language of cyberbullying . For definitions and examples from the dataset to the categories, see Appendix A.1. We build on the taxonomy suggested in Zampieri et al. (2019), which has been applied to datasets in several languages as well as in SemEval (Zampieri et al., 2020). While Parikh et al. (2019) provide a rich collection of sexism categories, multiple, overlapping labels do not fulfill the purpose of being easily understandable and applicable for annotators. The taxonomies in Anzovino et al. (2018) and Jha and Mamidi (2017) have proved their application to English, Italian and Spanish, and offer more general labels. Some labels from previous work were removed from the labeling scheme during the weekly discussions among authors and annotators, (for instance derailing ), because no instances of them were found in the data. During our analysis of misogyny in the Danish context (b), we became aware of the term neosex-ism. Neosexism is a concept defined in Tougas et al. (1999), and presents as the belief that women have already achieved equality, and that discrimination of women does not exist . Neosexism is based on covert sexist beliefs, which can go unnoticed, disappearing into the cultural norms. Those who consider themselves supporters of women's rights may maintain non-traditional gender roles, but also exhibit subtle sexist beliefs (Martinez et al., 2010). Sexism in Denmark appear to correlate with the modern sexism scale (Skewes et al., 2019; Tougas et al., 1995; Swim et al., 1995; Campbell et al., 1997). Neosexism was added to the taxonomy before annotation began, and as we will see in the analysis section, neosexism was the most common not sexual harassment misogyny neosexism discredit stereotypes & objectification benevolent sexism dominance disgrace/ humiliate women (Figure 1). Here follow some examples of neosexism from our dataset:  Resenting complaints about discrimination: I often feel that people have treated me better and spoken nicer to me because I was a girl, so I have a hard time taking it seriously when people think that women are so discriminated against in the Western world.  Questioning the existence of discrimination: Can you point to research showing that childbirth is the reason why mothers miss out on promotions?  Presenting men as victims: Classic. If it's a disadvantage for women it's the fault of society. If men, then it must be their own. Sexism thrives on the feminist wing. Neosexism is an implicit form of misogyny, which is reflected in annotation challenges summarised in section 5.5. In prior taxonomies, instances of neosexism would most likely have been assigned to the implicit appearances of misogynistic treatment (ii) (Guest et al., 2021) or perhaps not classified as misogyny at all. Neosexism is most closely related to the definition disrespectful actions, suggesting or stating that women should be controlled in some way, especially by men. This definition, however, does not describe the direct denial that misogyny exists. Without a distinct and explicit neosexism category, however, these phenomena may be mixed up or even ignored. taxonomies in abusive language while integrating context-related occurrences. A similar idea is demonstrated in Mulki and Ghanem (2021), adding damning as an occurrence of misogyny in an Arabic context. While most of previous research is done in English, these language-specific findings highlight the need for taxonomies that are flexible to different contexts, i.e. they are good representations of the world . Lastly, from an NLP point of view, languages with less resources for training data can profit further from transfer learning with similar labels, as demonstrated in Pamungkas et al. (2020) for misogyny detection. The final dataset contains 27.9K comments, of which 7.5K contain abusive language. Misogynistic posts comprise 7% of overall posts. Neosexism is by far the most frequently represented class with 1.3K tagged posts, while Discredit and Stereotype & objectification are present in 0.3K and 0.2K posts. Benevolent , Dominance , and Harrassment are tagged in between only 45 and 70 posts. Most posts tagged as abusive and/or containing misogyny are retrieved from searches on posts from public media profiles, see Table 3. Facebook and Twitter are equally represented, while Reddit is in the minority. Reddit posts were sampled from an available historical collection. and comments to these posts; keyw. = keyword/hashtag-search; popul. = most interactions. Frequencies of the words; kvinder' ( women ) and mnd' ( men ) were the highest, but these words did not represent strong polarities towards abusive and misogynistic content (Table 4). The word user' represents de-identified references to discussion participants (@USER). 5.4 Inter-Annotator Agreement (IAA) We measure IAA using the agreement between 3 annotators for each instance until round 3 (7k posts), and then sub-sampled data overlaps between 2 annotators. IAA is calculated through average label agreement at post level  for example if two annotators label two posts [abusive, untargeted] and [abusive, group targeted] the agreement would be 0.5. Our IAA during iterations of dataset construction ranged between 0.5 and 0.71. In the penultimate annotation round we saw a drop in agreement (Figure 2); this is attributed to a change in underlying text genre, moving to longer Reddit posts. 25% of disagreements about classifications were solved during discussions. Annotators had the opportunity to adjust their disagreed annotation in the first revision individually, which represents the remaining 75% (Table 5). The majority of disagreements were on subtask A, deciding whether the post was abusive or not. The final overall Fleiss' Kappa (Fleiss (1971)) for individual subtasks are: abusive/not: 0.58, targeted: 0.54, misogyny/not: 0.54. It is notable here that the dataset is significantly more skewed than prior work which upsampled to 1:1 class balances. Chance-corrected measurements are sensitive to agreement on rare categories and higher agreement is needed to reach reliability, as shown in Artstein and Poesio (2008). Based on the discussion rounds, the following types of posts were the most challenging to annotate: 1. Interpretation of the author's intention (irony, sarcasm, jokes, and questions) E.g. Haha! Virksomheder i Danmark: Vi anstter aldrig en kvinde igen... (Haha! Companies in Denmark: We will never hire a woman again ...) sexisme og seksuelt frisind er da vist ikke det samme? (I don't believe sexism and sexual liberalism are the same?) 2. Degree of abuse: Misrepresenting the truth to harm the subject or fact E.g. Han er en stor lgner (He is a big liar) 3. Hashtags: Meaning and usage of hashtags in relation to the context E.g. #nometoo 4. World knowledge required: Du siger at Frank bruger sin magt forkert men du bruger din til at brnde sa mange mnd pa balet ... (You say that Frank uses his power wrongly, but you use yours to throw so many men on the fire ... referring to a specific political topic.) 5. Quotes: re-posting or re-tweeting a quote gives limited information about the support or denial of the author 6. Jargon: receiver's perception I skal alle have et klap i masen herfra (You all get a pat on the behind from me) Handling these was an iterative process of raising cases for revision in the discussion rounds, formulating the issue, and providing documentation. We added the status and, where applicable, outcome from these cases to the guidelines. We also added explanations of hashtags and definitions of unclear identities, like the media, as a company. For quotes without declaration of rejection or support, we agreed to label them as not abusive, since the motivation of re-posting is not clear. Lastly, we provide a classification baseline: For misogyny and abusive language, the BERT model from Devlin et al. (2019) proved to be a robust architecture for cross-domain (Swamy et al., 2019) and cross-lingual (Pamungkas et al., 2020; Mulki and Ghanem, 2021) transfer. We use therefore multilingual BERT ('bert-base-multilingual-un cased') for general language understanding in Danish, fine-tuned on our dataset. Model: We follow the suggested parameters from Mosbach et al. (2020) for fine-tuning (learn-ing rate 2e-5, weight decay 0.01, AdamW optimizer without bias correction). Class imbalance is handled by weighted sampling and data split for train/test 80/20. Experiments are conducted with batch size 32 using Tesla V100 GPU. Preprocessing: Our initial pre-processing of the unstrucutured posts included converting emojis to text, url replacement, limit @USER and punctuation occurrences and adding special tokens for upper case letters adopted from Ahn et al. (2020). Classification: Since the effect of applying multi-task-learning might not conditionally improve performance (Mulki and Ghanem, 2021), the classification is evaluated on a subset of the dataset for each subtask (see Table 6) including all posts of the target label (e.g. misogyny) and stratified sampling of the non-target classes (e.g. for non-misogynistic: abusive and non-abusive posts) with 10k posts for each experiment. Results are reported when the model reached stabilized per class f1 scores for all classes on the test set (  0.01/20). The results indicate the expected challenge of accurately predicting less-represented classes and generalizing to unseen data. Analysing False Positives and False Negatives on the misogyny detection task, we cannot recognise noticeable correlations with other abusive forms and disagreements/ difficult cases from the annotation task. The goal was to ensure, first, a suf-ficient amount of misogynistic content and, secondly, mitigation of biases stemming from a uniform dataset. Surprisingly, topic sampling unearthed a higher density of misogynistic content than targeted keyword search (Table 3). While researching platforms, we noticed the limited presence of Danish for publicly available men-dominated fora (e.g. gaming forums such as DotA2 and extremist plaftorms such as Gab (Kennedy et al., 2018)). This, as well as limitations of platform APIs caused a narrow data selection. Often, non-privileged languages can gain from cross-language transfer learning. We experimented with translating misogynistic posts from Fersini et al. (2018) to Danish, using translation services, and thereby augment the minority class data. Translation services did not provide a sampling alternative. Additionally, as discovered by Anzovino et al. (2018), misogynistic content seems to vary with culture. This makes total text corrected label corrected out 960 877 224 48 Table 7: Translating IberEval posts EN to DA language-specific investigations important, both for the sake of quality of automatic detection systems, as well as for cultural discovery and investigation. Table 7 shows results of post-translation manual correction by annotators (all fluent in English). Reflections on annotation process Using just seven annotators has the disadvantage that one is unlikely to achieve as broad a range of annotator profiles as, for instance, through crowdsourcing. However, during annotation and weekly discussions, we saw clear benefits from having a small annotator group with different backgrounds and intense training. While annotation quality cannot be measured by IAA alone, the time for debate clar-ified taxonomy items, gave thorough guidelines, and increased the likelihood of correct annotations. The latter reflects the quality of the final dataset, while the former two indicate that the taxonomy and codebook are likely useful for other researchers analysing and processing online misogyny. The semi-open development of the taxonomy and frequent discussions allowed the detection neosexism as an implicit form of misogyny. Future research in taxonomies of misogyny could consider including distinctions between active/passive misogyny, as suggested by Anzovino et al. (2018) as well as other sub-phenomena. In the resulting dataset, we saw a strong representation of neosexism . Whether this is a specific cultural phenomenon for Danish, or indicative of general online behaviour, is not clear. The use of unified taxonomies in research affords the possibility to test the codebook guidelines iteratively. We include a short version of the guidelines in the appendix; the original document consists of seventeen pages. In a feedback survey following the annotation work, most of the annotators described that during the process, they used the guidelines primarily for revision in case they felt unsure how to label the post. To make the annotation more intuitively clear for annotators, we suggest reconsidering documentation tools and their accessibility for annotators. Guidelines are crucial for handling linguistic challenges, and well-documented decisions about them serve to create comparable research on detecting online misogyny across languages and dataset. In this work, we have documented the construction of a dataset for training systems for automatic detection of online misogyny. We also present the resulting dataset of misogyny in Danish social media, Bajer, including class balance, word counts, and baseline as an indicator. This dataset is available for research purposes upon request. The objective of this research was to explore the design of an annotation process which would result in a high quality dataset, and which was transparent and useful for other researchers. Our approach was to recruit and train a diverse group of annotators and build a taxonomy and codebook through collaborative and iterative annotator-involved discussions. The annotators reached good agreement, indicating that the taxonomy and codebook were understandable and useful. However, to rigorously evaluate the quality of the dataset and the performance of models that build on it, the models should be evaluated in practice with different text types and languages, as well as compared and combined with models trained on different datasets, i.e. Guest et al. (2021). Because online misogyny is a sensitive and precarious subject, we also propose that the performance of automatic detection models should be evaluated with use of qualitative methods (Inie and Derczynski, 2021), bringing humans into the loop. As we found through our continuous discussions, online abuse can present in surprising forms, for instance the denial that misogyny exists. The necessary integration of knowledge and concepts from relevant fields, e.g. social science, into NLP research is only really possible through thorough human participation and discussion. This research was supported by the IT University of Copenhagen, Computer Science for internal funding on Abusive Language Detection; and the Independent Research Fund Denmark under project 9131-00131B, Verif-AI. We thank our annotators Nina Schler Nrgaard, Tamana Saidi, Jonas Joachim Kofoed, Freja Birk, Cecilia Andersen, Ul-rik Dolzyk, Im Sofie Skak and Rania M. Tawfik. We are also grateful for discussions with Debora Nozza, Elisabetta Fersini and Tracie Farrell. Usernames and discussion participant/author names are replaced with a token @USER value. Annotators were presented with the text of the post and no author information. Posts that could not be interpreted by annotators because of missing background information were excluded. We only gathered public posts. All further information about dataset creation is included in the main body of the paper above.
abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective objective method objective other abstain abstain other other other other method other other other other other other other other other other other other abstain other other abstain other other other other other other other method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain other abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain objective abstain result abstain other other other abstain abstain abstain abstain abstain