As offensive content has become pervasive in social media, there has been much research in identifying potentially offensive messages. However, previous work on this topic did not consider the problem as a whole, but rather focused on detecting very specific types of offensive content, e.g., hate speech, cyberbulling, or cyber-aggression. In contrast, here we target several different kinds of offensive content. In particular, we model the task hierarchically, identifying the type and the target of offensive messages in social media. For this purpose, we complied the Offensive Language Identification Dataset (OLID), a new dataset with tweets annotated for offensive content using a fine-grained three-layer annotation scheme, which we make publicly available. We discuss the main similarities and differences between OLID and pre-existing datasets for hate speech identification, aggression detection, and similar tasks. We further experiment with and we compare the performance of different machine learning models on OLID. Offensive content has become pervasive in social media and thus a serious concern for government organizations, online communities, and social media platforms. One of the most common strategies to tackle the problem is to train systems capable of recognizing offensive content, which can then be deleted or set aside for human moderation. In the last few years, there have been several studies on the application of computational methods to deal with this problem. Prior work has studied offensive language in Twitter (Xu et al., 2012; Burnap and Williams, 2015; Davidson et al., 2017; Wiegand et al., 2018), Wikipedia comments, 1 and Facebook posts (Kumar et al., 2018). Previous studies have looked into different aspects of offensive language such as the use of abusive language (Nobata et al., 2016; Mubarak et al., 2017), (cyber-)aggression (Kumar et al., 2018), (cyber-)bullying (Xu et al., 2012; Dadvar et al., 2013), toxic comments 1 , hate speech (Kwok and Wang, 2013; Djuric et al., 2015; Burnap and Williams, 2015; Davidson et al., 2017; Malmasi and Zampieri, 2017, 2018), and offensive language (Wiegand et al., 2018). Recently, Waseem et al. (2017) analyzed the similarities between different approaches proposed in previous work and argued that there was a need for a typology that differentiates between whether the (abusive) language is directed towards a specific individual or entity, or towards a generalized group, and whether the abusive content is explicit or implicit. Wiegand et al. (2018) further applied this idea to German tweets. They experimented with a task on detecting offensive vs. nonoffensive tweets, and also with a second task on further sub-classifying the offensive tweets as profanity, insult, or abuse. However, to the best of our knowledge, no prior work has explored the target of the offensive language, which might be important in many scenarios, e.g., when studying hate speech with respect to a specific target. Below, we aim at bridging this gap. More generally, in this paper, we expand on the above ideas by proposing a novel three-level hierarchical annotation schema that encompasses the following three general categories: A: Offensive Language Detection B: Categorization of Offensive Language C: Offensive Language Target Identification Tweet A B C @USER He is so generous with his offers. We further use the above schema to annotate a large dataset of English tweets, which we make publicly available online. 2 The key contributions of this paper can be summarized as follows:  We propose a new three-level hierarchical annotation schema for abusive language detection and characterization. We apply the schema to create Offensive Language Identification Dataset (OLID) , a new large-scale dataset of English tweets with high-quality annotation of the target and type of offenses. We perform experiments on OLID using different machine learning models for each level of the annotation, thus setting important baselines to compare to in future work. While each of these sub-tasks tackles a particular type of abuse or offense, they share similar properties and the hierarchical annotation model proposed in this paper aims to capture this. Considering that, for example, an insult targeted at an individual is commonly known as cyberbulling and that insults targeted at a group are known as hate speech, we believe that OLID's use of a hierarchical annotation schema makes it a useful resource for various offensive language identification and characterization tasks. In the OLID dataset, we use a hierarchical annotation schema split into three levels to distinguish between whether the language is offensive or not (A), its type (B), and its target (C). Each level is described in more detail in the following subsections and examples are shown in Table 1. 2 The data can be downloaded from the following address: http://scholar.harvard.edu/malmasi/olid 2.1 Level A: Offensive language Detection Level A discriminates between the following types of tweets:  Not Offensive (NOT): Posts that do not contain offense or profanity;  Offensive (OFF): Posts containing any form of non-acceptable language (profanity) or a targeted offense, which can be veiled or direct. This includes insults, threats, and posts containing profane language or swear words. Level B categorizes the type of offense: Targeted Insult (TIN): Posts containing in-sult/threat to an individual, a group, or others;  Untargeted (UNT): Posts containing non-targeted profanity and swearing. Posts with general profanity are not targeted, but they contain non-acceptable language. 2.3 Level C: Offensive Language Target Identification Level C categorizes the targets of insults/threats:  Individual (IND): Posts targeting an individual. This can be a famous person, a named individual or an unnamed participant in the conversation. Insults and threats targeted at individuals are often defined as cyberbulling. Group (GRP): Posts targeting a group of people considered as a unity due to the same ethnicity, gender or sexual orientation, political affiliation, religious belief, or other common characteristic. Many of the insults and threats targeted at a group correspond to what is commonly understood as hate speech. Other (OTH) The target of these offensive posts does not belong to any of the previous two categories (e.g., an organization, a situation, an event, or an issue). We retrieved the examples in OLID from Twitter using its API and searching for keywords and constructions that are often included in offensive messages, such as she is' or to:BreitBartNews' 3 . The full list of keywords we used is shown in Table 2. We first carried out a round of trial annotation of 300 instances with six experts using nine keywords. The goal of the trial annotation was ( i ) to evaluate the proposed tagset, ( ii ) to evaluate the data retrieval method, and ( iii ) to create a gold standard with instances that could be used as test questions to ensure the quality of the annotators for the rest of the data, which was carried out using crowdsourcing. The keywords used in the trial annotation are shown in the first nine rows of Table 2. We included left ( @NewYorker ) and far-right ( @BreitBartNews ) news accounts because there tends to be political offense in the comments for such accounts. The keyword that resulted in the highest concentration of offensive content was the Twitter safe' filter, corresponding to tweets that were flagged as unsafe by Twitter (the -' sym-biol indicates not safe'). Since the vast majority of content on Twitter is not offensive, we tried different strategies to keep the distribution of offensive tweets at around 30% of the dataset. We excluded some keywords that were not high in offensive content during the trial annotation such as they are' and to:NewYorker'. Although he is' was poor in offensive content in the trial dataset (15%), we kept it as a keyword in order to avoid gender bias, and we found that in the full dataset it was more offensive (32.4%). The trial keywords that we ultimately decided to exclude due to low percentage of offensive tweets are shown in the top portion of Table 2. We computed Fleiss' kappa on the trial dataset for the five annotators on 21 of the tweets. The value was .83 for Layer A (OFF vs. NOT) indicating high agreement. As to normalization and anonymization, we did not store any user meta-data or Twitter IDs, and we substituted the URLs and the Twitter mentions by placeholders. During the full annotation task, we decided to search for more political keywords as they tend to be richer in offensive content. Thus, we sampled our full dataset, so that 50% of the tweets come from political keywords, and the other 50% come from non-political keywords. Within these two groups, tweets were evenly sampled for the keywords. In addition to gun control', and to:BreitbartNews' used during the trial annotation, four new political keywords were used to col-lect tweets for the full dataset: MAGA', antifa', conservatives', and liberals'. The breakdown of keywords and their offensive content in the full dataset is shown in the bottom of Table 2. We follow prior work in related areas (Burnap and Williams, 2015; Davidson et al., 2017) and we annotate our data using crowdsourcing. We used Figure Eight 4 and we ensured data quality by ( i ) only hiring annotators who were experienced in the platform, and ( ii ) using test questions to discard annotations by individuals who did not reach a certain threshold. Each instance in the dataset was annotated by multiple annotators and inter-annotator agreement was calculated at the end. We first acquired two annotations for each instance. In the case of disagreement, we requested a third annotation, and we then took a majority vote. The annotators were asked to label each tweet at all three levels of the annotation scheme, and we considered there to be agreement only when the annotators agreed on the labels for all levels. Approximately 60% of the time, the two annotators agreed, and thus no additional annotation was needed. A third annotation was requested for the rest of the tweets; there was no instance when more than three annotations were needed. The breakdown of the data into training and testing for the labels from each level is shown in Table 3. It is worth noting that one of the key challenges we observed when collecting for OLID was producing a dataset containing a sufficient number of instances for each class. This is particularly evident in the sizes for Subtasks B and C. Other studies also had this issue when collecting similar datasets. For example, in (Davidson et al., 2017), only 5% of the tweets were considered hate speech by the majority of the annotators, and in (Burnap and Williams, 2015) only 11.6% of the examples were labeled as hate speech. SVM Our simplest machine learning model is a linear SVM trained on word unigrams. SVMs have achieved state-of-the-art results for many text classification tasks (Zampieri et al., 2018). BiLSTM We also experiment with a bidirectional Long Short-Term-Memory (BiLSTM) model, which we adapted from a pre-existing model for sentiment analysis (Rasooli et al., 2018). The model consists of ( i ) an input embedding layer, ( ii ) a bidirectional LSTM layer, and ( iii ) an average pooling layer of input features. The concatenation of the LSTM layer and the average pooling layer is further passed through a dense layer, whose output is ultimately passed through a softmax to produce the final prediction. We set two input channels for the input embedding layers: pre-trained FastText embeddings (Bo-janowski et al., 2017), as well as updatable embeddings learned by the model during training. CNN Finally, we experiment with a Convolutional Neural Network (CNN) model based on the architecture of (Kim, 2014), and using the same multi-channel inputs as the above BiLSTM. Our models are trained on the training dataset, and evaluated by predicting the labels for the held-out test set. As the label distribution is highly im-balanced (see Table 3), we evaluate and we compare the performance of the different models using macro-averaged F1-score. We further report per-class Precision (P), Recall (R), and F1-score (F1), and weighted average. Finally, we compare the performance of the models against simple majority and minority class baselines. The performance on discriminating between offensive (OFF) and non-offensive (NOT) posts is reported in Table 4. We can see that all models perform significantly better than chance, with the neural models performing substantially better than the SVM. The CNN outperforms the RNN model, achieving a macro-F1 score of 0.80. In this set of experiments, the models were trained to discriminate between targeted insults and threats (TIN) and untargeted (UNT) offenses, which generally refer to profanity. The results are shown in Table 5. We can see that the CNN performs better than the BiLSTM, with a macro-F1 score of 0.69. Note that all models perform better at identifying TIN compared to UNT. The results for the offensive target identification experiment are shown in Table 6. Here the models were trained to distinguish between three targets: a group (GRP), an individual (IND), or others (OTH). We can see that all three models achieved similar results, far surpassing the random baselines, with a slight performance edge for the neural models. The performance of all models for the OTH class is 0, which can be explained by two factors. First, unlike the two other classes, OTH is a heterogeneous collection of targets. It includes offensive tweets targeted at organizations, situations, events, etc., thus making it more challenging for models to learn discriminative properties for this class. Second, there are fewer training instances for this class compared to the other two: there are only 395 instances for OTH vs. 1,075 for GRP and 2,407 for IND. We presented OLID, a new dataset with annotation of type and target of offensive language. It is the official dataset of the shared task SemEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) (Zampieri et al., 2019). 5 In OffensEval, each annotation level in OLID is an independent sub-task. To the best of our knowledge, this is the first dataset to contain annotation of type and target of offenses in social media, and it opens interesting research directions. We further presented baseline experiments using SVMs and neural networks, which have shown that this is a challenging, yet doable task. In future work, we would like to make a cross-corpus comparison of OLID vs. datasets annotated for similar tasks such as aggression identification (Kumar et al., 2018) and hate speech detection (Davidson et al., 2017). We further plan to create similar datasets for other languages, following OLID's hierarchical annotation scheme. We would like to thank the anonymous NAACL reviewers for their valuable suggestions and Nikola Ljubesic for the feedback provided. This research presented was partially supported by an ERAS fellowship awarded to Marcos Zampieri by the University of Wolverhampton.
abstain abstain objective method objective abstain method abstain abstain abstain abstain other abstain abstain abstain objective objective objective method objective objective method objective method method abstain other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain objective abstain abstain objective result abstain method other other