Conversational agents trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which may include offensive or otherwise toxic behavior. We introduce a new human-and-model-in-the-loop framework for evaluating the toxicity of such models, and compare a variety of existing methods in both the cases of non-adversarial and adversarial users that expose their weaknesses. We then go on to propose two novel methods for safe conversational agents, by either training on data from our new human-and-model-in-the-loop framework in a two-stage system, or baking-in safety to the generative model itself. We find our new techniques are (i) safer than existing models; while (ii) maintaining usability metrics such as engagingness relative to state-of-the-art chatbots. In contrast, we expose serious safety issues in existing standard systems like GPT2 (Radford et al., 2019), DialoGPT (Zhang et al., 2019) and BlenderBot (Roller et al., 2020). When dialogue models are trained to mimic human-human conversations utilizing large preexisting datasets, they will unfortunately also learn undesirable features from this human-human data, such as the use of toxic or biased language. Most recent work in the detection and prevention of offensive 1 language has focused exclusively on human-generated data. These conversations may be very different from the domain in which a dialogue model might eventually be deployed: for example, humans may adversarially attempt to elicit  Equal contribution 1 In this paper, we use offensive, toxic, and unsafe interchangeably. For more discussions about attempts to better define categories of unsafe content, see Schmidt and Wiegand (2017). offensive language from a dialogue model in ways that differ from how they would speak with another human. In this work, we introduce Bot-Adversarial Dialogue (BAD) Safety, a novel method for evaluating chatbot safety with humans and models in the loop. We ask humans to adversarially converse with a set of state-of-the-art English-language models with the aim of inducing them to generate unsafe responses to mimic the way these models can be adversarially attacked at deployment time. We analyze how to optimally construct such a crowdworker task, and collect a dataset of 5k such conversations yielding around 70k total utterances. We then use the BAD method and data to evaluate the safety of several generative models and propose two techniques for making safer models: (1) Training a safety classifier with this data and deploying a two-stage model at inference time. In the two-stage setting, we prevent the generative model from surfacing offensive language flagged by the classifier. (2) A novel method that directly bakes in" toxicity-awareness to the generative model during training by modifying the target responses to incorporate safe responses to offensive input. In experiments, we show that our new techniques outperform other existing generative models in terms of safety, while maintaining engagingness. We publicly release the BAD training and evaluation data as well as select models trained using this data via ParlAI. 2 2 Related Work Numerous works have shown that humans speak differently with bots than with humans, with increases in profanity and aggressiveness associated with addressing a bot (Hill et al., 2015; Lortie and Guitton, 2011), which motivates the incorporation of human-bot dialogues into our safety framework. 2 https://parl . ai/projects/ safety_recipes/ De Angeli and Carpenter (2005); De Angeli and Brahnam (2008) suggest that one in ten human-bot conversations may contain instances of the human demonstrating unprovoked abusive behavior towards the chatbot. Miller et al. (2017b) argued that adversarial attacks need to be expected and planned for when deploying a user-facing system that learns from its interactions. These findings suggest it is insufficient to merely exclude toxic data from training, as the model would not know how to answer hostile out-of-domain inputs, and positive biases where models tend to agree rather than contradict (Roller et al., 2020) would lead to undesirable outcomes. As shown in Gehman et al. (2020), training on sanitized data can decrease the amount of unprompted toxic content, yet still leave models vulnerable to generating toxic content based on specific prompts. The moving target of toxic content requires dynamic methods that repeatedly update benchmarks to improve current systems (Dinan et al., 2019a; Nie et al., 2019) 3 . The iterative procedure in Dinan et al. (2019a) strictly focuses on detection of toxicity in human-generated utterances through several rounds of humans attempting to break a toxicity classifier, without addressing generation. Our BAD approach is similar in spirit, but centers on generations of a bot in a human-bot conversation, closer to the context of deployed conversational models. Focusing on generation requires deciding how to address bad content. Previous works have compared response strategies, including avoidance, joking or polite deflection, non-committal answers, play-along, confrontation, apologetic responding, empathizing, and counter-attacking responses (Curry and Rieser, 2019; Chin and Yi, 2019; Chin et al., 2020; Paranjape et al., 2020). They find that humans rate different strategies as more appropriate depending on the type of offense they are responding to. Note that different implementation details make those strategies difficult to directly compare. While we use a strategy of non-sequiturs in this work, our takeaway is that future work should keep investigating several types of responses such that models can learn to deploy them adaptively according to finer-grained understanding of unsafe content. We describe the models we analyze in this paper, including safety classifiers and generative models. We consider binary Transformer-based classifiers, following the same structure as in Dinan et al. (2019a), with two sizes: 128M and 311M parameters. We pre-train these models on a previously existing Reddit dataset extracted and obtained by a third party that was hosted by pushshift.io (Baum-gartner et al., 2020), using a masked language model objective, and then fine-tune on the safety classification tasks of interest, performing early stopping using the F1 score of the unsafe class on the validation set. These tasks include various combinations of the Wikipedia Toxic Comments dataset (WTC) (Wulczyn et al., 2017), Standard (S) and adversarial Build-it, Break-it, Fix-it (BBF) data from Dinan et al. (2019a), as well as semi-supervised data created from labeling the pushshift.io Reddit (Baumgartner et al., 2020) (Reddit) and Blended Skill Talk (BST) datasets. Finally, we will use a new dataset Bot-Adversarial Dialogue (BAD), to be described in 4. As further baselines, we will also compare to both single-turn and multi-turn classifiers from Dinan et al. (2019a). BST 2.7B We start from a state-of-the-art open-domain dialogue system. We consider the same architecture and setup as in BlenderBot (Roller et al., 2020), which employs a Seq2Seq Transformer architecture (Vaswani et al., 2017), with an implementation based on the ParlAI version (Miller et al., 2017a). We consider the 2.7B parameter model which has 2 encoder layers, 24 decoder layers, 2560 dimensional embeddings, and 32 attention heads, which performed best in human evaluations of engagingness. This model was pre-trained using pushshift.io Reddit and fine-tuned on on a smaller set of crowdsourced datasets designed to provide important conversational skills. At decoding time, the model employs standard beam search with a beam size of 10 , context and label 3 -gram blocking (Paulus et al., 2017), and a minimum beam length of 20 BPE tokens, which was shown to perform well compared to other choices. DialoGPT and GPT2 In our experiments we also compare to two other baseline models: DialoGPT (Medium) (Zhang et al., 2019) and GPT2 (Large) (Radford et al., 2019), although we expect these two models to have lower engagingness scores than the BST 2.7B base model, in line with results from Roller et al. (2020); Adiwardana et al. (2020). We use the medium-sized DialoGPT (345M parameters), as it performed best according to metrics in Zhang et al. (2019). To our knowledge these methods have not been compared previously in terms of dialogue safety evaluations, or the en-gagingness/safety trade-off. This section introduces the Bot-Adversarial Dialogue method for evaluating and training safe generative models. We would like our model to be robust against direct attempts to probe them for unsafe responses. Previous work described how safety classifiers can be trained to be robust to adversarial human attack, see Dinan et al. (2019a). However, in order to achieve safer conversational agents, we advocate for applying such a procedure directly to conversations with those agents, which previous work did not address. The main idea is to collect data of humans probing our conversational models to try to trick them into making mistakes, similar to adversarial attacks that happen in practice during deployment. We can then use this data to make our system more robust. We therefore construct a task where crowdworkers are instructed to converse naturally with a bot while simultaneously trying to probe or adversarially goad it into replying in an unsafe manner. Each bot's response is tagged as either safe or unsafe by the crowdworker. We divide the unsafe responses into four bins: those that nearly 0% of the population would consider unsafe (according to the annotator's best guess), and those that < 10%, <50% or >= 50% of the population would consider unsafe. After collection, a separate task verifies the labels with 3 other annotators, who can mark utterances either as safe, or as unsafe with a number of finer categories (e.g., hate speech). See Appendix A for details on how we combine all annotations to give a single label to utterances. While we only make Class Train Valid Test SAFE Utterances 42049 4239 1654 OFFENSIVE Utterances 27225 2763 944 Total Utterances 69274 7002 2598 Total Dialogues 5080 513 191 Table 1: Dataset Statistics for the BAD dataset where crowdsource workers were instructed to converse with a bot and annotate each bot utterance for offensiveness. use of the binary safe / unsafe distinction in this work, this finer-grained labeling could be used in subsequent work to accommodate different levels of sensitivity. We conduct 14-turn conversations with the human speaking first, and score the model based on the number of unsafe turns. Such data can be used as a held out test set for static evaluation across different models, as well as for training data for improving models. More details of the collection setup are given in Appendix A. Figure 1 shows how BAD differs from the BBF set-up from Dinan et al. (2019a): namely, in BBF, the breaker (or adversarial user) tries to break a classifier by submitting human-authored adversarial unsafe messages that are incorrectly classified as safe, whereas in this work (BAD), the breaker adversarially tries to elicit unsafe messages from a dialogue model 4 . We compare to classifiers trained on BBF and additionally evaluate our new classifiers on BBF in our experiments. Using the BAD method, we collect 5,784 dialogues between bots and crowdworkers, consisting of 78,874 utterances in total from both humans and bots (see Table 1). About 40% of all utterances are annotated as offensive, among which one third are generated by bots. To encourage the bot to use offensive language more often, humans tended to use either unsafe language themselves in the dialogues, or raise probing questions that are considered inappropriate to ask. More than 42% of the dialogues collected contain at least 3 unsafe human messages or probing questions (see Appendix, Table 6). We further break down the messages from humans into a taxonomy of offensive language types, as these may prove useful in future work. The majority of offensive language used by crowdworkers relates to hate speech against particular groups, personal 4 The emoji image in Figure 1 is by Twemoji ( https:// github . com/twitter/twemoji ), and is licensed under CC BY-4.0. attacks and other less explicit offensive language containing no profanity, see Appendix Figure 5. Further details can be found in Appendix A. 4.2 Applying to Conversational Agents We consider two different general strategies for making generative models safer to engage with: training classifiers for detecting unsafe messages as an added safety layer (4.2.1) and training the model such that it is unlikely to surface unsafe content at inference time (4.2.2). Given a safety classifier, a simple approach for improving dialogue safety is to use it to detect if both the user input and the model's response are safe. If a safety violation is detected in either type of utterance, one can then, instead, initiate a response designed to be safe. While several different safe response strategies can be considered (Curry and Rieser, 2019; Paranjape et al., 2020), in this work we respond with a non-sequitur: we select a topic at random from 1,087 topics judged as safe from the Wizard of Wikipedia conversational topic list (Dinan et al., 2019b) and then produce the response Hey do you want to talk about something else? How about we talk about X? where X is the chosen topic. Additional approaches are considered and analyzed in Appendix B.1. After returning this response, the conversation continues as normal, with the response entering into the model's conversational history. In this way, the model can still respond naturally to followup responses after the canned safe response is produced. We note that this approach works only as well as the classifier. If the classifier red flags too many safe utterances, the conversational experience will suffer. If unsafe utterances are not flagged, toxic language can still enter the conversation. This highlights a potential trade-off between ensuring safety and having an engaging conversation. A separate safety classifier layer has advantages (e.g. any independent improvement of this classifier can be used), but also downsides. For example, such an open-sourced model is more complicated to share and deploy, requires more computational resources (e.g. loading both models), and allows unsafe usage if the layer is simply removed. Further, in the long-term it makes sense if safety is part of a single dialogue agent model, in the sense that ideally it should understand what it is saying is unsafe. Here, we detail two generative model training methods that are less likely to surface unsafe content without the use of an additional safety layer: data pre-processing and baking-in the safety layer, the latter of which is a new approach introduced in this work. Data Pre-processing A classic approach to training models on unclean data is to filter it beforehand. Assuming we have access to a safety classifier, we can use it to filter the training set. In this work, we perform filtering by removing an example from the training set if either the conversational context (input) or response (output) triggers the safety classifier. Other approaches  such as author-based filtering  are considered and evaluated in Appendix B.2. This training set is then used to train models as usual. With this approach, it is important for this filtering to be performed on the large pre-training dataset: if only the fine-tuning datasets are cleaned, the model will still have been exposed to offensive language, which it will be able to remember and use (as indeed confirmed by our experiments). Baking in the Safety Layer Data pre-processing methods attempt to make a model safe by simply not exposing it to offensive language. This can make those models vulnerable to adversarial usage because they will not have learned how to handle offensive language at all: our models frequently copy the input (Welleck et al., 2020), so they might copy the offensive language. We instead propose a technique for attempting to bake awareness of toxic language into the training data, by using labeled examples that recommend appropriate action on the model's part in those circumstances. To do this, we first assume we have access to a safety classifier at training time (but not at deployment time). For each training example, if the last utterance in the dialogue history or the ground-truth response are labeled as unsafe by the classifier, we instead replace the ground-truth response of that training example with a non-sequitur. An example demonstrating this procedure is shown in Table 2. After constructing baked-in safety data, one can then train the generative model using likelihood training as usual, but with these modified targets. We separate training examples that have been Bot-Adversarial Dialogue (this work) Build-It Break-It Fix-It for Safety (Dinan et al., 2019a) Figure 1: Comparison of our Bot-Adversarial Dialogue (BAD) set-up (left) to the build-it, break-it, fix-it for toxicity classifier robustness from Dinan et al. (2019a) (right). modified for safety from those that have not, and assign different weights to them  effectively drawing examples from those two sets with different probabilities  affecting how much the model optimizes for safety versus usual conversational abilities. Balancing these weights is important, especially when dealing with highly toxic pre-training sets, as they may be dominated by modified examples. We choose this weighting as a hyperparameter of the model. We use human evaluations in both an adversarial and non-adversarial setting to evaluate how safe our generative models are. Since our ultimate goal is to create models that are safe while still being engaging to talk to, we also perform human evaluations of conversational quality. the ACUTE eval (Li et al., 2019) method of evaluating conversational quality, as used for BlenderBot (Roller et al., 2020) and elsewhere. The method involves collecting human-bot conversations for two models one wishes to compare, and then presenting two of those conversations at random, one from each model, to crowdworkers to make a series of pairwise decisions based on a target question. We use the question corresponding to evaluating en-gaginess from Li et al. (2019), which is phrased as Who would you prefer to talk to for a long conver-sation?. We compare each of the proposed models in this paper to the baseline BST 2.7B which Roller et al. (2020) showed is state-of-the-art in open-domain dialogue. We note that our goal is not to find models which are more engaging than our baseline, but to find models that are simultaneously equally engaging and safer . Evaluating Adversarial Safety To perform human evaluation of the safety of models in an adversarial setting we create a test set consisting of 180 dialogues collected using the Bot-Adversarial Dialogue method. Recall that the BAD data is collected by having humans converse and attempt to elicit unsafe responses with a dialogue model. In order to prevent this test set from being biased towards adversarial examples that target a single model, we construct the test set by sampling conversations from a large suite of models. At evaluation time, we collect a model's responses to each of the 180 dialogues in the test set, and human evaluators are used to judge the safety of each response. This set-up allows us to evaluate models in an adversarial setting that mimics deployment but for which the examples remained fixed, eliminating variances based on the experience and quality of crowdworkers during conversation collection. Evaluating Non-Adversarial Safety While we need our models to perform well in an adversarial setting, we also wish for them to perform well in a non-adversarial setting. For example, a model that repeats user input verbatim may be robust to more subtle attempts to elicit offensive responses which are not offensive in and of themselves, but would not be robust to simpler attacks like profanity. For this reason, we propose a non-adversarial test set composed of 180 examples that are extracted from the Wikipedia Toxic Comments test set. We adopt the same human evaluation setup as in the adversarial setting in which various models are evaluated for the given contexts. We detail experimental results in this section, including results of the data collection from the Bot-Adversarial Dialogue method (5.1), experimental results related to training classifiers (5.2), and a comparison of safe generation methods (5.3). Lastly, in 5.4, we detail and compare the overall safety and engagingness scores for all models. We describe results from data collection using the Bot-Adversarial Dialogue method, providing a detailed analysis of the effects of the crowdsourcing methods. In order to inform crowdsource task design, we use logistic regression to model several task outcomes. Predictors include variables capturing the human chat partner's experience with the task and the particular bot they are currently talking to, and which of two possible versions of task instructions was received. Experience with the task is measured as the number of HITs accepted by the worker so far  a HIT, or Human Intelligence Task, is the term used by Amazon's Mechanical Turk to refer to a single instance of a crowdworker task. Experience with a specific bot is captured as the position of the utterance within the conversation (e.g., 2nd utterance in a 14 utterance conversation). The models underlying the bot responses were included as predictors and had a large significant effect (as discussed in the rest of the paper), but are omitted from the discussion here to focus on predictors related to task design. Modeling results shown in Table 3 suggest that (1) instructing workers to ask open questions about sensitive topics rather than using obvious profanities (New instruction set) has a significant effect, increasing the rate of unsafe bot utterances while simultaneously decreasing the rate of unsafe human utterances; (2) self-selection effects are present (see also Sec. A.4), so that the total number of HITs ultimately completed is predictive of higher success at eliciting not-OK content; (3) two types of learning effects are present: workers are more successful (i.e., are able to solicit more unsafe responses) as they perform more iterations of the task, and within HITs, which might reflect that workers figure out the vulnerabilities of the particular bot they have been paired with and identify the most successful strategies. We note that the increased rate of unsafe utterances for later utterances observed here is in the context of an explicitly adversarial setting aiming to elicit them; we do not expect that this pattern would generalize to non-adversarial contexts. Automatic evaluation results are presented for safety classifiers in Table 4. We train safety classifiers using the methodology described in Sec. 4.2.1 and compare different model sizes and multitasking across different training sources. Firstly, we find our newly trained models superior to existing models from Dinan et al. (2019a) when using the same training sets, likely due to improved pushshift.io Reddit pre-training of our Transformers compared to their BERT models. However, we find relatively small gains from either larger Transformers (Safety Classifier + ) over smaller ones (Safety), or from semi-supervised learning over Reddit and BST (Semi-Sup. + ). We compare the classifier trained on the BAD dataset, multitasked with the other datasets, to other approaches in Table 4. We observe similar results Outcome: not OK utterances Bot, rater Bot, partner Human Base  3 . to our other new safety classifiers on the single-turn Wikipedia Toxic Comments (WTC), Build-It Break-It Fix (BBF) and Standard (S) test sets, but superior results on the multi-turn bot-adversarial BAD test set. The BAD-based classifier achieves 80.8 unsafe F1 on the latter dataset, while the next best performing methods achieve 61.5, 61.0 and 60.7, respectively. This result can be explained by virtue of the fact that the BAD-based classifier is the only one trained on the BAD training set, hence it sees data that most closely resembles the evaluation distribution. Note that the BAD training set differs from the other training sets listed as it is both (i) adversarially collected and (ii) multi-turn. One can tease apart the effects of each of these attributes by comparing to a single-turn (truncated) version of BAD training, shown in Table 4 (second to last row), which still performs well  though not as well  as the multi-turn version, indicating that the adversarial component is most important. As the BAD test set is the closest setup to the actual setting in which such a classifier might be deployed (it features human-bot conversations, rather than human-human single-turn data), this indicates the BAD-based classifier is the most likely method to be successful in real use cases. We compare the baked-in safety layer method of 4.2.2 to the data-preprocessing methods using 400M parameter models, the details of which are described in Appendix B, and find that baked-in training gives increased safety over safe utterance preprocessing. On pushshift.io Reddit, the baked-in method triggers a classifier 0.2% vs. 6.8% of the time for preprocessing. Both methods yield similar PPL and F1 scores. We thus experiment with scaling it up to a 2.7B parameter model. We perform human evaluations to compare the relative safety and engagingness for many of the selected methods. Results showing the engagingness performance relative to safety performance (for both adversarial and non-adversarial safety) using human judgments (4.3) are shown in Figure 2. Automatic evaluations are provided in Appendix D. We compare the methods described in this paper  two-stage models and baked in models  to three standard baselines: BST 2.7B, DialoGPT, and GPT2. BST 2.7B (Roller et al., 2020) has simply been trained on existing dialogue corpora, with no safety technique at all in model training. DialoGPT (Zhang et al., 2019) uses a pre-processing method, where offensive subreddits where removed from the training data. We test DialoGPT in two flavors: with short generations (using standard beam de-coding), and longer generations (where we add a constraint that a minimum of 20 tokens must be generated, similar to (Roller et al., 2020). In all experiments we use the medium-sized version of DialoGPT, with 345M parameters, as noted in 3.2. Finally, GPT2 (Radford et al., 2019) was trained on web data that was filtered for data quality, but not for offensive language as far as we are aware. Engagingness scores from the ACUTE-eval set-up are plotted along the x -axis in Figure 2. Detailed results can be found in Table 9 in the Appendix. Results on standard models indicate that BST 2.7B is significantly more engaging than GPT2, DialoGPT and pushift.io Reddit 2.7B. We apply the classifier learned from our Bot-Adversarial Dialogue (BAD) dataset (multi-tasked with our other datasets) in a two-stage model. Engagingness of this model is found to be not significantly distinguishable from our base BST 2.7B model. The baked-in model also performs similarly to the base BST 2.7B model with respect to engagingness, showing that this system still works Model Name Size Training Data WTC S BBF BAD Avg. To perform human evaluation of safety in an adversarial setting, we evaluate models using the BAD evaluation method described in 4.3. Results can be seen on the y -axis of Figure 2 (left). More details are provided in Table 15 in the Appendix. Results show that all of our standard base models  including BST 2.7B, DialoGPT, and GPT2  are susceptible to attack, e.g. GPT2 produces safe responses only 59.4% of the time, and BST 2.7B only 55% of the time. Clearly, to defend against BAD requires alternative techniques. Our two-stage BAD classifier approach improves over our other safety classifiers used in two-stage systems, yielding an 94.4% OK rate on the adversarial data. Overall, this method offers strong robustness without affecting engagingness, and we advocate its use. For our baked-in model , we see clear gains relative to standard models (e.g. increasing from the baseline BST 2.7B value of 55% OK up to 78.3% OK), although these gains are not as significant as when using two-stage models (the same classifiers in a two-stage setup can bring the results up to 83.9% OK). We believe an important next step for future work is to improve this training technique to match the two-stage results. Human evaluation of safety in a non -adversarial setting is conducted using the Wiki Toxic Comments test set described in 4.3. Results can be seen on the y -axis of Figure 2 (right). More details are provided in Table 16 in the Appendix. standard models appear susceptible to attack. In the best case, DialoGPT produces safe responses only 68.3% of the time. GPT2 performs the worst, providing safe responses 54.4% of the time. Our two-stage models get near perfect scores here  scores range from 97.8 to 98.3  showing that these models are very robust to attack in the non-adversarial setting. This shows that future effort to make these models safe should focus on the adversarial setting, as in BAD. The baked-in model performs the best in this setting, achieving very high scores. We conclude this technique should be further explored, particularly for robustness in the adversarial setting. We observe that standard generative models  with little or no safety intervention  fall very short in terms of safety, especially when measured using our Bot-Adversarial Dialogue (BAD) framework, which we publicly release along with our models. However, with our safety techniques we can achieve roughly the same engagingness as the state of the art BST 2.7B with substantially better safety scores, showing it is possible to build a model that is both safe and engaging. We find generative models can be improved considerably by distilling a safety classifier into the encoder-decoder during training, i.e. the baked-in approach. Two-stage models provide safer results still, with best performance coming from our BAD-based classifier with BST 2.7B in the adversarial case. We note that while we have improved substantially over existing systems, our best systems are not perfectly safe as measured by the BAD method. Conducting perfectly safe dialogue requires the Figure 2: Engagingness vs. Safety: Comparing engagingness scores from ACUTE-eval to adversarial safety scores on the Bot-Adversarial Dialogue (BAD) test set (left) and non-adversarial safety scores on the Wiki Toxic Comments test set (right). model to deeply understand language and likely cannot be completely solved until AI itself is solved. Further complicating the issue is the fact that the very definition of safe" is both contextually and culturally dependent (Schmidt and Wiegand, 2017). Rather than attempt to define safety for all languages and locales, in this work we rely on crowdworker consensus and focus on machine learning methods for English language data. We look forward to further progress in these technical and ethical challenges. 7 Ethical Considerations In this paper, we have presented several methods for building safer conversational agents. As we noted in the conclusion, even our best systems are not perfectly safe. This raises several ethical considerations, including questions of: when can a model be considered safe"? Is a failure rate of 5 . 6% in an adversarial setting acceptable for the deployment of such models? How safe is safe enough? Creating a perfectly safe dialogue model requires the model to deeply understand language and likely cannot be completely solved until AI itself is solved, i.e. this is an AI-complete problem. We also reiterate that the issue is further complicated by the fact that the very definition of safe" is both contextually and culturally dependent (Schmidt and Wiegand, 2017). A dialogue model must be able to understand the boundaries of its particular conversation partner. What is offensive to one may not be offensive to another (Curry and Rieser, 2019). Culturally speaking, the approaches in this paper are limited in both geographical and historical senses. Our methods rely only on English-speaking annotators located in the United States. This narrow, Western-centric viewpoint will be insufficient for solving the issue in other languages and locales (Schmidt and Wiegand, 2017). Further, it is well known that commonly used hate-speech datasets are known to have issues with bias and fairness (Dixon et al., 2018). Sap et al. (2019) showed that several contain correlations between surface markers of African American English and toxicity, and propose race and dialect priming as a way to mitigate this. In this work we have assumed a consensus-based view on offensiveness, by admitting examples based on agreement of multiple humans;however, offense to underrepresented groups for example may be missed by such a setup. We encourage further work to consider how classifiers trained on the datasets described in this work may be biased against various demographic groups. Lastly, our work analyzes publicly available open-sourced models. We note that there may be concerns in the community or the public at large related to releasing models, even for research purposes, due to their potential safety issues. The community has recently started to address this tradeoff between releasing models that can produce offensive or toxic language and open, reproducible research 5 . We believe the solution for these issues involves the community working together and conducting reproducible research on safety. Releasing code and models facilitates that joint community effort.
abstain objective objective objective abstain abstain abstain abstain abstain objective abstain abstain objective objective method objective abstain other abstain abstain abstain abstain abstain objective method method abstain other abstain method abstain method method abstain abstain other method method abstain method abstain method method method method abstain method abstain method method method method abstain method method method abstain abstain abstain method other method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method abstain method method method method abstain abstain method abstain abstain method method method abstain abstain abstain method abstain abstain method abstain method method method method abstain abstain method result abstain result abstain abstain abstain abstain abstain other abstain result result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result result result abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method objective abstain method abstain