We study the task of toxic spans detection, which concerns the detection of the spans that make a text toxic, when detecting such spans is possible. We introduce a dataset for this task, TOXICSPANS , which we release publicly. By experimenting with several methods, we show that sequence labeling models perform best. Moreover, methods that add generic rationale extraction mechanisms on top of classifiers trained to predict if a post is toxic or not are also surprisingly promising. Finally, we use TOXICSPANS and systems trained on it, to provide further analysis of state-of-the-art toxic to non-toxic transfer systems, as well as of human performance on that latter task. Our work highlights challenges in finer toxicity detection and mitigation. In social media and online fora, toxic content can be defined as rude, disrespectful, or unreasonable posts that would make users want to leave the conversation (Borkan et al., 2019). Although several toxicity detection datasets (Wulczyn et al., 2017; Borkan et al., 2019) and models (Schmidt and Wie-gand, 2017; Pavlopoulos et al., 2017c; Zampieri et al., 2019) exist, most of them classify whole posts, without identifying the specific spans that make a text toxic . But highlighting such toxic spans can assist human moderators (e.g., news portal moderators) who often deal with lengthy comments, and who prefer attribution instead of a system-generated unexplained toxicity score per post. Locating toxic spans within a text is thus a major step towards successful semi-automated moderation and healthier online discussions. toxic spans, called TOXICSPANS . 1 We discuss how it was created and propose an evaluation framework for toxic spans detection. We consider methods that (i) perform sequence labeling (tag words) or (ii) rely on an attentional binary classifier to predict if a post is toxic or not, then invoke its attention at inference time to obtain toxic spans as in rationale extraction. The latter approach allows leveraging larger existing training datasets, which provide gold labels indicating which posts are toxic or not, without providing gold toxic span annotations. Although sequence labeling performed overall better, the binary attentional classifier performed surprisingly well too, despite having been trained on data without span annotations. We then study some characteristics of supervised and self-supervised toxic-to-civil transfer models (Laugier et al., 2021) by comparing them on several datasets, including a recently released parallel toxic-to-civil dataset (Dementieva et al., 2021) and the new TOXICSPANS dataset. Using the latter, we introduce a measure to evaluate the elimination of explicit toxicity, and we use this measure to compare the behavior and performance of toxic-to-civil models. Lastly, by applying toxic span detection systems, we assess the performance of human crowdworkers on the toxic-to-civil task. Toxicity detection systems (Schmidt and Wiegand, 2017; Pavlopoulos et al., 2017c; Zampieri et al., 2019) are typically trained on datasets annotated at the post level (a text is annotated as toxic or not) (Wulczyn et al., 2017; Borkan et al., 2019). Our work differs from general toxicity detection 1 Our code and dataset are publicly available at https: //github.com/ipavlopoulos/toxic_spans with a CC0 licence. Part of the dataset was also used in the SemEval-2021 Task 5 (Pavlopoulos et al., 2021). in that we detect toxic spans , instead of assigning toxicity labels to entire texts. Toxic spans detection can be seen as a case of attribution or rationale extraction (Li et al., 2016; Ribeiro et al., 2016; Lei et al., 2016; Zhang et al., 2021; Jain et al., 2020; DeYoung et al., 2020), but specifically for toxic posts, a task that has never been considered in general toxicity detection before. Detecting spans, instead of entire posts, was recently also considered in propaganda (Martino et al., 2020) and hate speech detection (Mathew et al., 2021). Although the ground truth type is similar (spans), propaganda detection is a different task from ours. Hate speech is a particular type of toxicity (Borkan et al., 2019), which can be tackled by more general toxicity detectors (Van Aken et al., 2018), but not the other way round; i.e., we address a broader problem. This probably explains why a pattern-matching baseline, based on the data of Mathew et al. (2021), achieved only slightly better results than a random baseline on our dataset. Suggesting civil rephrases of posts found to be toxic (Nogueira dos Santos et al., 2018; Laugier et al., 2021) is the next step towards healthier online discussions, and can be viewed as style transfer (Shen et al., 2017; Fu et al., 2018; Lample et al., 2019). We show how toxic spans detection can contribute in the assessment of toxic-to-civil transfer, linking the two tasks together for the first time. We used posts (comments) from the publicly available Civil Comments dataset (Borkan et al., 2019), which already provides whole-post toxicity annotations. We followed the toxicity definition that was used in Civil Comments, i.e., we use toxic' as an umbrella term that covers abusive language phenomena, such as insults, hate speech, identity attack, or profanity. This definition of toxicity has been used extensively in previous work (Hosseini et al., 2017; Van Aken et al., 2018; Karan and na-jder, 2019; Han and Tsvetkov, 2020; Pavlopoulos et al., 2020). We asked crowd annotators to highlight the spans that constitute anything that is rude, disrespectful, or unreasonable that would make someone want to leave a conversation. Besides toxicity our annotators were also asked to select a subtype for each highlighted span, choosing between insult, threat, identity-based attack, profane/obscene, or other toxicity. Asking the annotators to also select a category was intended as a priming exercise to increase their engagement, but it may have also helped them align their notions of toxicity further, increasing inter-annotator agreement. For the purposes of our experiments, we collapsed all the subtypes into a single toxic class, and we did not study them further; but the subtypes are included in the new dataset we release. Annotation From the original Civil Comments dataset (1.2M posts), we retained only posts that had been found toxic by at least half of the crowd-raters. This left approx. 30k toxic posts. We selected a random 11k subset of the 30k posts for toxic spans annotation. We used the crowd-annotation platform of Appen. 2 We employed three crowd-raters per post, all of whom were warned for explicit content. Raters were selected from the smallest group of the most experienced and accurate contributors. The raters were asked to mark the toxic word sequences (spans) of each post by highlighting each toxic span on their screen. For each post, the dataset includes the spans of all three raters. If the raters believed a post was not actually toxic, or that the entire post would have to be annotated, they were instructed to select appropriate tick-boxes in the interface, without highlighting any span. The tick-boxes were separate and the dataset shows when (if) any of the two were ticked. Hence, when no toxic spans are provided (for a particular post by a particular rater), it is clear if the rater thought that the post was not actually toxic, or that the entire post would have to be annotated. message being conveyed may be inherently toxic (e.g., a sarcastic post indirectly claiming that people of a particular origin are inferior) and, hence, it may be difficult to attribute the toxicity of those posts to particular spans. In such cases, the posts may end up having no toxic span annotations, according to the guidelines given to the annotators; see the last post of Table 1 for an example. In other cases, however, it is easier to identify particular spans (possibly multiple per post) that make a post toxic, and these toxic spans often cover only a small part of the post (see Table 1 for examples). Agreement We measured inter-annotator agreement on 87 randomly selected posts of our dataset, using 5 crowd-annotators per post in this case. We calculated the mean pairwise (for a pair of annotators) Cohen's kappa per post, using character offsets as instances being classified as toxic (included in a toxic span) or non-toxic; we then averaged over the posts. Although our dataset contains only posts found toxic by at least half of the original crowd-raters, only 31 of the 87 posts were found toxic by all five of our annotators, and 51 were found toxic by the majority of our annotators; this is an indicator of the well-known subjectivity of toxicity detection. On the 31, 51, and 87 posts, the average kappa score was 65%, 55%, 48%, respectively, indicating that when the raters agree (at least by majority) about the toxicity of the post, there is also reasonable agreement regarding the toxic spans. Note that the toxic spans are typically short. This leads to class imbalance (most offsets are marked as non-toxic), increases agreement by chance (on the non-toxic offsets), and leads to low kappa scores (kappa adjusts for chance agreement). Another reason behind this modest (compared to other tasks) inter-annotator agreement is the inherent subjectivity of deciding if a post is toxic or not. Our kappa score is in fact slightly higher than in previous work on toxicity detection, classifying posts as toxic or not (Sap et al., 2020; Pavlopoulos et al., 2017a), and in that sense our inter-annotator agreement can be seen as an improvement. Ground truth To obtain the ground truth of our dataset, we averaged the labels per character of the annotators per post. We used the following process: for each post t , first we mapped each annotated span of each rater to its character offsets. We then assigned a toxicity score to each character offset of t , computed as the fraction of raters who annotated that character offset as toxic (included it in their toxic spans). We retained only character offsets with toxicity scores higher than 50%; i.e., at least two raters must have included each character offset in their spans. Table 1 shows examples. The dataset TOXICSPANS contains the 11,035 posts we annotated for toxic spans. The unique posts are actually 11,006, since a few were duplicates and were removed in subsequent experiments. A few other posts were used as quiz questions to check the reliability of candidate annotators and were also discarded in subsequent experiments. Exploratory analysis Although we instructed the crowd-raters to click the appropriate tick-box and not highlight any span when the whole post would have to be highlighted, the ground truth of 34 out of the 11k posts covers the entire post. However, 14 out of the 34 posts are single-word texts, while the other posts are very short (Appendix A shows more details); it seems that in very short posts the raters sometimes did not realize they ended up highlighting the entire post. Furthermore, about 5k of the 11k posts have an empty ground truth set of toxic character offsets (as in the last post of Table 1), even though all the posts of our dataset had been found toxic by the original raters. This is partly due to the fact that we include in the ground truth only character offsets that were included in the toxic spans of the majority of our annotators. It also confirms it is not always possible to attribute (at least not by consensus) the toxicity of a post to particular toxic spans. In almost all posts, the ground truth covers less than half of the post; and in the vast majority, less than 20% of the post. A dense toxic span of a post is a maximal sequence of contiguous toxic characters. There exist posts with more than one dense toxic span, but most posts include only one. Table 2 provides further statistics. For the newly introduced toxic spans detection task, we evaluate systems in terms of F 1 score, as in the work of Da San Martino et al. (2019). Given a test post t , let system A i return a set S tA i of character offsets, for parts of the post found to be toxic. Let S t G be the character offsets of the ground truth annotations of t . We compute the F 1 score of system A i with respect to the ground truth G for post t : F t 1 ( A i , G ) = 2  P t ( A i , G )  R t ( A i , G ) P t ( A i , G ) + R t ( A i , G ) (1) 3723 Mean Min Max Post length 208.14 4 1,000 Dense toxic span length 7.01 3 87 # Dense toxic spans 0.58 0 8 Table 2: TOXICSPANS statistics. If S tG is empty for some post t (no gold spans are given for t ), we set F t 1 ( A i , G ) = 1 if S tA i is also empty, and F t 1 ( A i , G ) = 0 otherwise. We average F t 1 ( A i , G ) over all test posts t to obtain a single score for system A i . We use F 1 as the main evaluation measure in experiments reported below. TRAIN-MATCH , is a simple lookup-based model that classifies as toxic any tokens encountered inside toxic spans of the training data. HATE-MATCH operates similarly, but the lookup is within the hate-ful/offensive spans of the data of Mathew et al. (2021). A naive baseline, RAND-SEQ , randomly classifies tokens as toxic or not. Toxic spans detection can be seen as sequence labeling (tagging words). As a baseline of this kind, we employ SPACY ' S Convolutional Neural Network, which is pre-trained for tagging, parsing, entity recognition (Honnibal and Montani, 2017). We call this model CNN-SEQ and fine-tune it on dense toxic spans, treated as entities'. We also train a bidirectional LSTM ( BILSTM-SEQ ), 3 and fine-tune BERT (Devlin et al., 2019) and SPANBERT (Joshi et al., 2020) for toxic spans ( BERTSEQ , SPAN-BERT-SEQ ). 4 These methods require training data manually annotated with toxic spans. We trained binary classifiers to predict the toxicity label of each post, and we employed attention as a rationale extraction mechanism at inference to obtain toxic spans, an approach Pavlopoulos et al. (2017b) found to work reasonably well in toxicity detection. 5 We experimented with two classifiers: 3 We used the probabilistic ground truth for training and mean square error as the loss function of BILSTM-SEQ , which yielded best results in preliminary experiments. 4 More details can be found in the Appendix A.3. a BILSTM with deep self-attention as in the work of Pavlopoulos et al. (2017b), but training with a regression objective and probabilistic labels following D'Sa et al. (2020) and Wulczyn et al. (2017); and BERT with a dense layer and sigmoid on the [ CLS ] embedding. To detect toxic spans, we used the attention scores of the BILSTM and the attention scores from the heads of BERT 's last layer averaged over the heads, respectively. In both cases, we obtain a sequence of binary decisions (toxic, nontoxic) for the tokens of the post (inherited by their character offsets) by using a probability threshold (tuned on development data) applied to the attention scores. We refer to these two attention-based rationale extraction methods as BILSTM + ARE and BERT + ARE , respectively. These methods require training posts annotated only with toxicity labels per post (no toxic span annotations). We used a 5-fold Monte Carlo cross-validation (5 random training/development/test splits) on the 11k posts of TOXICSPANS . In each fold, we use 10% of the data for testing, 10% for development, and 80% for training. In ARE -based methods, which rely on an underlying classifier to predict if a post is toxic or not, the classifier is trained on the training part of the fold (which contains only toxic posts, ignoring the toxic span annotations) and a randomly selected but not in toxicity detection. See also Wiegreffe and Pinter (2019), Kobayashi et al. (2020), Ferrando and Costa-juss (2021) for a broader discussion of attention as an explainability mechanism. equal number of non-toxic posts from Civil Comments that are not included in our dataset. When measuring the (binary) classification performance of the underlying classifier, the classifier is evaluated on a new equally balanced test set of 3k randomly sampled unseen posts from Civil Comments. Both look-up methods ( TRAIN-MATCH , HATEMATCH ) outperform the random baseline (Table 3). However, TRAIN-MATCH performs much better, which agrees with our hypothesis that toxicity detection is a broader problem than hate speech detection. Both look-up methods are outperformed by the sequence labeling models (-SEQ ), especially SPAN-BERT-SEQ , which is pre-trained to predict spans. These results show that the tokens of toxic spans are context-dependent and their meaning is not captured well by context-unaware look-up lexicons. An error analysis of the best-performing SPAN-BERT-SEQ showed that mistakes include both false negatives (e.g., incorrectly returning an empty span, 1st row of Table 4) and false positives (2nd and 3rd row). BERT + ARE performs worse than BILSTM + ARE , despite the fact that the underlying BERT classifier is much better ( ROC AUC 96.1%) at separating toxic from nontoxic posts than the underlying BILSTM (90.9%). Interestingly, the BILSTM binary toxicity classifier with the attention-based toxic span detection mechanism (Pavlopoulos et al., 2017b) is close in performance with BILSTM-SEQ , despite the fact that the latter is directly trained on toxic span annotations, whereas the former is trained with binary post-level annotations only (toxic, non-toxic post ). Several large datasets with post-level toxicity annotations are publicly available (Pavlopoulos et al., 2019). Therefore, attribution-based toxic span detectors, such as BILSTM + ARE , can in principle perform even better if the underlying binary classifier is trained on a larger existing dataset. To investigate this, we increased the training set of the underlying BILSTM classifier of BILSTM + ARE . We added to the training set of each cross-validation fold 80k further toxic and non-toxic posts (still equally balanced, without toxic spans) from the dataset of Borkan et al. (2019), excluding posts used in TOXICSPANS . The ROC AUC score of the underlying BILSTM (in the task of separating toxic from nontoxic posts) improved from 90.9% to 94.2%, and the F 1 score of BILSTM + ARE (in toxic spans detection) improved from 57.7% to 58.8%, almost reaching the performance of BILSTM-SEQ . 6 7 Toxic spans in toxic-to-civil transfer As shown in Section 6, a toxic span detection method can be used to highlight toxic parts of a post, to assist, for instance, human moderators. The new TOXICSPANS dataset and toxic span detection methods, however, can assist in more ways. This section describes how we combined the new dataset and the best-performing toxic span detector ( SPAN-BERT-SEQ ) to show how they can be useful in toxic-to-civil text transfer (Nogueira dos Santos et al., 2018; Laugier et al., 2021). In the context of detoxifying comments to nudge users towards healthier conversations online, this task aims at suggesting civil rephrasings of toxic posts. More specifically, we study the following research question: Can TOXICSPANS data and toxic span detectors be used to assess the mitigation of explicit toxicity in toxic-to-civil transfer? To answer this question, we proceeded in two ways: (i) evaluating the transfer of toxic spans in system -detoxified posts, and (ii) studying any remaining toxic spans in human -detoxified posts. We first compare the performance of two toxic-to-civil transfer models, CAE-T 5 and SED-T 5, both based on the T 5 transformer encoder-decoder architecture (Raffel et al., 2019); they both fine-tune the weights of the same pre-trained model, namely T 5-large. CAE-T 5 (Laugier et al., 2021) is a self-supervised Conditional Auto-Encoder, fine-tuned on a large non-parallel ( NP ) dataset based on preprocessed posts from the Civil Comments ( CC ) dataset, the dataset (with post-level annotations) that TOXICSPANS was also based on. SED-T 5 is a 6 Appendix A reports results for less added data. Supervised Encoder-Decoder; we fine-tuned it on a smaller parallel ( P ) dataset created by Dementieva et al. (2021), consisting of pairs of comments: a toxic comment and a detoxified paraphrase written by a crowdworker. Table 5 summarizes statistics of the two datasets ( P , NP ) and highlights a trade-off between the level of supervision and number of samples: there is a 1:40 ratio between toxic comments in P (direct supervision, parallel data) and NP (indirect supervision, no parallel data). Table 6 shows our experimental results. Following Laugier et al. (2021), we report accuracy ( ACC ), perplexity ( PPL ), similarity ( SIM ), and the geometric mean ( GM ) of ACC , 1 / PPL , SIM . Accuracy measures the rate of successful transfers from toxic to civil, and computes the fraction of posts whose civil version is classified as non-toxic by a BERT toxicity classifier; we used the BERT -based toxicity classifier of Laugier et al. (2021). Perplexity is used here as a measure of fluency and is computed with GPT -2 (Radford et al., 2019). Similarity measures content preservation between the original toxic text and its system-rephrased civil version (selfSIM ) or the gold (hu-man) civil rephrasing (refSIM , only for P ); in both cases, it is computed as the cosine similarity between the single-vector representations of the two texts, produced by the universal sentence encoder of Cer et al. (2018). As can be seen in Table 6, CAE-T 5 has better aggregated results (higher GM ) than SED-T 5 in all three datasets, which are due to lower perplexity and (in NP and TOXICSPANS ) higher accuracy. However, SED-T 5 learned to preserve content better (higher SIM in all three datasets), because of the parallel data ( P , with gold rephrases) it was trained on. By contrast, CAE-T 5 was trained without parallel data ( NP ) using a cycle-consistency loss, which leads to more frequent hallucinations of content that was not present in the original post (Laugier et al., 2021). These hallucinations may also help CAE-T 5 obtain better perplexity scores, by gener-Evaluation Dataset Metric CAE-T 5 SED-T 5 Non-Parallel ( NP ) ACC  75.0% 52.2% ACC 2  83.4% 67.3% PPL  5.2 11.8 selfSIM  70.0% 87.9% GM (self)  0.466 0.338 ACC 3  86.7% 64.1% ACC 4  83.2% 59.5% Parallel ( P ) ACC  94.3% 94.3% ACC 2  94.7% 94.3% PPL  9.1 38.3 refSIM  27.6 % 65.3% selfSIM  32.6 % 65.6% GM (ref)  0.306 0.252 GM (self)  0.323 0.252 ACC 3  98.8% 94.3% ACC 4  94.7% 91.9% TOXICSPANS ACC  92.9% 65.6% ACC 2  92.5% 63.7% PPL  7.2 24.9 selfSIM  34.5% 82.1% GM (self)  0.355 0.279 ACC 3  96.9% 62.0% ACC 4  92.0 % 54.7% Table 6: Automatic evaluation scores of CAE-T 5 (trained on NP 's training subset) and SED-T 5 (trained on P 's training subset), when the test sets are from NP , P , and TOXICSPANS . ACC 2, ACC 3, ACC 4 also consider toxic spans (Section 7.2). ating fluent civil rephrases' that do not preserve, however, the original semantics. Also, although the general trends are similar in all three datasets ( SED-T 5 preserves content better, CAE-T 5 is better in perplexity and GM ), there are several differences too across the three datasets. For example, CAET 5 is much better than SED-T 5 in accuracy (posts detoxified) on NP and TOXICSPANS , but both systems have the same accuracy on P ; and the scores of the systems vary a lot across the three datasets. These considerations motivated us to seek ways to further analyse the behavior of toxic-to-civil transfer models. TOXICSPANS and toxic span detectors are an opportunity to move towards this direction, by studying how well transfer models cope with explicit toxicity , i.e., spans that can be explicitly pointed to as sources of toxicity. We leave for future work the flip side of this study, i.e., studying cases where transfer models rephrase spans not explicitly marked (by toxic span detectors or human annotators) as explicitly toxic. Recall that the accuracy ( ACC ) scores of Table 6 measure the percentage of toxic posts that the transfer models ( CAE-T 5, SED-T 5) rephrased to forms that a ( BERT -based) toxicity classifier considered non-toxic. One could question, however, if it is possible (even for humans) to produce a civil rephrase 3726 of a toxic post when it is impossible to point to particular spans of the post that cause its toxicity (as in the last post of Table 1). Detoxifying posts of this kind may constitute a mission impossible for most models (possibly even for humans); the only way to produce a non-toxic rephrase' may be to change the original post beyond recognition, which may be rewarding systems like CAE-T 5 that often hallucinate in their rephrases, as already discussed. Hence, it makes sense to focus on posts that contain explicit toxic spans, marked by human annotators (for TOXICSPANS ) or our best toxic span detector ( SPAN-BERT-SEQ ). Using these toxic spans, we define three additional variants of accuracy: ACC 2 is the same as ACC , but ignores (in its denominator) posts that do not contain at least one toxic span; ACC 3 also considers (in its denominator) only posts that contained at least one toxic span, but computes the fraction of these posts that had all of their toxic spans rephrased (even partly) by the transfer model; ACC 4 is a stricter version of ACC 3 that requires the posts to also be judged non-toxic by the ( BERT based) toxicity classifier. Table 6 shows that restricting ACC to consider only posts with at least one toxic post ( ACC 2) substantially improves the performance of both models on the NP dataset, indicating that it contains many mission impossible' instances (posts with no toxic spans) that the original ACC considers. By contrast, switching from ACC to ACC 2 leads to mostly negligible changes on the P and TOXICSPANS datasets, which is in accordance with the fact that they contain fewer posts with no toxic spans (11.5% and 48.7%, respectively, compared to 67.4% for NP ). Another interesting observation is that ACC 4 is always substantially lower than ACC 3 (for both systems, on all three datasets), indicating that the models often successfully detect toxic spans and try to rephrase them, but the rephrases are still toxic, at least according to the toxicity classifier. In this experiment, we wished to study the extent to which humans rephrase known toxic spans, when asked to produce civil rephrases of toxic posts. We used the P dataset, the only one of the three considered that contains human rephrases. 7 Since P does not contain gold toxic spans, we again employed SPAN-BERT-SEQ to add toxic spans to the source posts and retained only the 1,354 (out of 2,778 in 7 We used all the P data, since no training was involved. total) source-target pairs of posts with at least one toxic span in their source post. 8 In all but 6 of the 1,354 posts, the humans have rephrased (in the gold target post they provided) all the toxic spans of the source post. The 6 posts were mainly cases where the human changed the context to mitigate toxicity, while retaining the original toxic span. For example,  he's not that stupid  became  he's not stupid  (original toxic span shown in bold); in this case removing the that' from the context arguably makes the post less offensive. Overall, we conclude that humans did rephrase almost all cases of explicit toxicity in the toxic posts they were given. We also applied SPAN-BERT-SEQ to the gold target (rephrased) posts that the humans provided to check if any explicit toxicity remained or was introduced by the rephrases. This flagged 93 gold target posts as comprising at least one toxic span. A manual inspection of the 93 posts revealed that they fall in two main categories. The first category comprises cases where a toxic span of the source post was rephrased, but the rephrase might not be considered totally civil; e.g.,  how freaking narcissistic do you have to be?  became  how narcissistic do you have to be? , where SPAN-BERT-SEQ marked the narcissistic' of the rephrase as a toxic span. The second category comprises cases where SPANBERT-SEQ produced false positives; e.g., the source post  most of the information is total garbage  became  most of the information is totally useless , but SPAN-BERT-SEQ marked (arguably incorrectly) useless' as a toxic span. We also applied the BERT -based text toxicity classifier of Laugier et al. (2021) to the 2,778 posts of the P dataset, dividing them in two sets: posts that comprised at least one toxic span detected by SPAN-BERT-SEQ (1,354 posts with explicit toxicity) and the rest (implicit toxicity). The BERT -based toxicity classifier considered more toxic (higher average toxicity score) the 1,354 posts of the first set compared to the second one, i.e., it was more confident that the posts of the first set (explicit toxicity) were toxic, as one might expect. By resampling 1,000 subsets (of 50 posts each) from the two sets, we confirmed that this is a statistically significant difference ( P = 0 . 001 ). The difference of the average predicted toxicity score between the two sets 8 The most frequent spans were sh*t', st*p*d', f*ck'. is 14% (from 0.94 down to 0.80). The posts we annotated for toxic spans were extracted from an already heavily studied public do-main benchmark dataset (Civil Comments) that has been examined by thousands of teams in a Kag-gle competition, 9 and that has been cited in over 50 academic publications. The Civil Comments dataset was filtered to remove any potential personally identifiable information before it was released. Our annotation cost was $21,089 for 59,486 judgements, paying $0.30 per item. All raters were warned for the explicit content of the job and only high accuracy raters were selected (70+%), based on performance on quiz questions. The most common countries of origin of our crowd-annotators were Venezuela and USA (Fig. 6 in Appendix A.1). In the contributor satisfaction survey, 51 participants gave an overall task rating of 3.6/5.0, with pay and test question fairness rated slightly higher than ease of job and clarity of instructions. We note that it is more difficult and costly (ap-proximately 3 times more) to manually annotate toxic spans, instead of just labeling entire posts as toxic or not. This is why we also explored adding rationale extraction components on top of toxicity classifiers trained on existing much larger datasets. We showed that BILSTM + ARE has the potential to reach the performance of BILSTM-SEQ , which is important for future work aiming to build toxic span detectors without any toxic span annotations in the training data. This may be particularly useful in low-resourced languages with limited resources for text toxicity (Zampieri et al., 2020). Having two separate systems, one for toxicity detection and one for toxic spans identification, is more easily compatible with existing deployed toxicity detectors. One can simply add a component for toxic spans at the end of a pipeline for toxicity detection, and the new component would be invoked only when toxicity would be detected, leaving the rest of the existing pipeline unchanged. Since the vast majority of posts in real-world applications is non-toxic (Borkan et al., 2019), this pipeline approach would only increase the computational load for the relatively few posts classified as toxic. Using only toxic posts in this study was also a way to simplify this first approach to toxic spans detection, assuming an oracle system achieved the first step 9 shorturl.at/hqEJ3 (deciding which posts are toxic). However, we note that future work could study adding non-toxic posts to our dataset and requiring systems to first detect toxic posts, then extract toxic spans for toxic posts. A direct comparison (in terms of size) of TOXICSPANS with other existing toxicity datasets is only possible if one focuses on the toxic class, typically the minority one, since our dataset contains only toxic posts. By adding non-toxic posts, much larger versions of our dataset can be compiled, of sizes similar to those of existing previous datasets (that provide post-level annotations only). Hence, our TOXICSPANS dataset is accessible with the following versions: First, only toxic posts included (11,006 posts), which is the version we discuss in this work. Second, the previous version will be augmented with the same number of randomly selected non-toxic Civil Comments posts. Third, a version similar to the previous one, but where the ratio of toxic to non-toxic posts will be 1:40 to be closer to that of real-world datasets (325,499 posts). As shown in Section 7, the TOXICSPANS dataset and toxic span detectors can also help study and evaluate explicit toxicity removal when rephrasing toxic posts to be civil. In this case, toxic spans can be used to get a better understanding of how toxic-to-civil models operate, by showing the toxic spans and their context, along with their rephrases. The toxic span detection systems we consider are trained (the sequence-labeling ones) and tested (all systems) on posts with binary ground-truth character offset labels (toxic or not), reflecting the majority opinion of the annotators (Section 3). This runs the risk of ignoring the opinions of minorities, who may also be minorities among crowd-annotators. To address this issue, we also release the toxic spans of all the annotators and the pseudonymous rater identities, not just the spans that reflect the majority opinion, to allow different label binarisation strategies and further studies. Toxic span detection systems are intended to assist the decision making of moderators, not to replace moderators. When they operate correctly, systems of this kind are expected to ease decision making (reject/accept a post). Incorrect results could be of two types; toxic spans that were not highlighted and non-toxic spans that were highlighted. Mistakes of both types, especially of the first one, may mislead a moderator working under time pressure. As with other content filtering systems (e.g., spam filters, phishing detectors), toxic span detectors may trigger an adversarial reaction of malicious users, who may study which types of toxic expressions evade the detectors (esp. publicly available ones) and may gradually start using more implicit toxic language (e.g., irony, false claims), which may be more difficult to detect. However, this is a danger that concerns any toxicity detection system, including systems that classify user content at the post level (without detecting toxic spans). We studied toxicity detection, which aims to identify the spans of a user post that make it toxic. Our work is the first of this kind in general toxicity detection. We constructed and released a dataset for the new task, along with baselines and models. Fine-tuning the SPAN-BERT sequence labelling model of Joshi et al. (2020), yielded the best results. A post-level BILSTM toxicity classifier that was combined with an attention-based attribution method, not trained on annotations at the span level, performed well for the task. By leveraging the dataset of posts annotated as toxic or nontoxic (without spans), we showed that this method can reach the performance of a BILSTM sequence labelling approach that was trained on the more costly toxic spans annotations. This result is particularly interesting for future work aiming to perform toxic spans detection by using only datasets with whole-post toxicity annotations. In a final experiment, we examined toxic-to-civil transfer, showing how toxic spans can help shed more light on this task too, by helping assess how well systems and humans address explicit toxicity. In future work we plan to study toxic span detection in multiple languages and in context-dependent toxic posts. We thank Lucas Dixon for discussions, insight, and useful comments. We also thank the anonymous reviewers for their comments. This research was funded in part by an unrestricted gift from Google.
method abstain result abstain method result abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain method method other other other abstain other other other method abstain other objective method method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain method method method abstain method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain result abstain result method other other other