Abstract Warning: this paper contains examples that may be offensive or upsetting. The social impact of natural language processing and its applications has received increasing attention. In this position paper, we focus on the problem of safety for end-to-end conversational AI. We survey the problem landscape therein, introducing a taxonomy of three observed phenomena: the INSTIGATOR , YEASAYER , and IMPOSTOR effects. We then empirically assess the extent to which current tools can measure these effects and current systems display them. We release these tools as part of a first aid kit (SAFETYKIT ) to quickly assess apparent safety concerns. Our results show that, while current tools are able to provide an estimate of the relative safety of systems in various settings, they still have several shortcomings. We suggest several future directions and discuss ethical considerations. Several recent studies discuss the potential harms and benefits of large language models (LLMs), e.g., Bender et al. (2021); Bommasani et al. (2021); Weidinger et al. (2021). Here, we turn our attention to neural conversational response generation models that are trained end-to-end on open-domain dialog data (E2E convAI). Examples include DialoGPT (Zhang et al., 2020b), Meena Bot (Adiwar-dana et al., 2020), and BlenderBot (Roller et al., 2021). In contrast to general generative or autoregressive LLMs, these specialized models are typically deployed in an interactive setting, i.e., conversing with a user. They are trained on large amounts of conversational data, for example Twitter, pushshift.io Reddit (Baumgartner et al., 2020), or OpenSubtitles dataset. Large neural models in general, and convAI models in particular, have been shown to replicate and even amplify negative, stereotypical, and derogatory associations in the data (Shah et al., 2020; Bender et al., 2021). In addition, neural LM generation is hard to control, although there are some first steps in this direction (Khalifa et al., 2021; Smith et al., 2020b). These two facts taken together can result in situations where convAI systems generate inappropriate content (Dinan et al., 2019; Xu et al., 2020), or respond inappropriately to offensive content (Cercas Curry and Rieser, 2018; Lee et al., 2019). Furthermore, recent research suggests that the anthropomorphic design of these systems (c.f. Abercrombie et al., 2021) correlates with increased instances of bullying behavior (Keijsers et al., 2021). This change in interaction style and the attribution of agency (Araujo, 2018) results in safety scenarios that are qualitatively different from LLMs: here, an inappropriate response might result in severe, or even life-threatening, consequences for the user (Bick-more et al., 2018). We summarize these issues resulting in potential harm under the term safety. In particular, we consider harmful system behavior that can lead to negative short-term impact, e.g., the user feeling insulted, and long-term harm, e.g., negative societal stereotypes being reinforced. We consider three safety-sensitive phenomena for conversational systems, which we refer to as: the INSTIGATOR , YEASAYER , and IMPOSTOR effects (see 2). We provide an in-depth discussion of the potential impact of these three scenarios and define them in the context of related work. We then empirically evaluate currently available tools for assessing the impact of E2E conversational AI models with respect to these phenomena. We perform detailed experiments and analyses of the tools therein using five popular conversational AI agents, release them in a open-source toolkit (SAFETYKIT ), and make recommendations for future use. We introduce a taxonomy of three safety-sensitive situations for E2E convAI models, summarized with examples in Table 1. We consider other issues related to the problem of safety for E2E convAI outside of the scope of this work; nevertheless, we briefly mention some of them in Appendix A. Note that this taxonomy has already inspired further work in this area (Sun et al., 2021). In the first scenario, a system generates harmful content, thereby directly instigating harm. One of the first and best-known examples is the Microsoft AI chatbot Tay, which was launched and subsequently shut down for producing offensive language (Miller et al., 2017). What is offensive content? Before diving into this phenomenon, we need to discuss the definition of offensive content, a well-studied subject in NLP. Ultimately, whether or not something is offensive is subjective, and several authors emphasize that any decisions (e.g., on classification or mitigation strategies) should respect community norms and language practices (Jurgens et al., 2019; Sap et al., 2019; Kiritchenko and Nejadgholi, 2020). Offensive content is therefore an umbrella term encompassing toxicity, hate speech, and abusive language (Fortuna et al., 2020). Khatri et al. (2018) define sensitive content more generally as offensive to people based on gender, demographic factors, culture, or religion. In addition to overtly offensive language, several works highlight the importance of including more subtle forms of abuse, such as implicit abuse and micro-aggressions (e.g., Jurgens et al., 2019; Caselli et al., 2020; Han and Tsvetkov, 2020). Thylstrup and Waseem (2020) caution that using binary labels in itself incurs the risk of reproducing inequalities. Detection of such problematic content online has attracted widespread attention in recent years, however, much of this focuses on human-produced content on social media platforms, such as Twitter (e.g. Waseem and Hovy, 2016; Wang et al., 2020; Zampieri et al., 2019, 2020), Facebook (Glava et al., 2020; Zampieri et al., 2020), or Reddit (Han and Tsvetkov, 2020; Zampieri et al., 2020). Notably less work exists for conversational systems; generally focusing on user input, rather than system-generated responses, (e.g. Dinan et al., 2019; Xu et al., 2020; Cercas Curry et al., 2021). Offensive system responses While less well-studied than human-generated offensive content, offensive content generated by the systems themselves  i.e., the INSTIGATOREFFECT  has been the subject of several recent works. Ram et al. (2017), for example, use keyword matching and machine learning methods to detect system responses that are profane, sexual, racially inflammatory, other hate speech, or violent. Zhang et al. (2020a) develop a hierarchical classification framework for malevolent responses in dialogues (al-though their data is from Twitter rather than human-agent conversations). And Xu et al. (2020) apply the same classifier they used for detection of unsafe user input to system responses. As in the case of Tay and more recently Luda (McCurry, 2021), 4114 conversational systems can also be vulnerable to adversarial prompts from users that elicit unsafe responses. Liu et al. (2020) demonstrate this by generating prompts that manipulated an E2E model to generate outputs containing offensive terms. Mitigation efforts A number of possible ways of mitigating offensive content generation in language models have been proposed. One possibility is to not expose the system to offensive content in its training data, e.g., by creating data filters (Ngo et al., 2021). However, in this scenario, models are still vulnerable to generating toxic content based on specific prompts (Gehman et al., 2020), even though the quantity of unprompted toxic content may decrease. Similarly, Cercas Curry and Rieser (2018) find that conversational E2E models trained on clean data can [still] be interpreted as flirtatious and sometimes react with counter-aggression when exposed to abuse from the user. Solaiman and Dennison (2021) find that, rather than filtering pre-training data, fine-tuning a language model on a small, curated dataset can be effective at limiting toxic generations. An alternative approach is to control the language generation process. Dathathri et al. (2019) use a simple classifier to guide a language model away from generation of toxic content. Liu et al. (2021) detoxify a language model's output by upweighting the probabilities of generating words considered unlikely by a second anti-expert model that models toxic language. Schick et al. (2021) propose something similar, but use instead the language model's own knowledge of toxic content to detect toxic generations in zero-shot manner. For our focus, the dialog domain, Xu et al. (2020) compare several train-time approaches for mitigating offensive generation: detoxifying the model's training set as a pre-processing step, and distilling knowledge of how to respond to offensive user by augmenting the training set. They also experiment with inference-time approaches, using both a two-stage set-up with a classifier in-the-loop and a token-blocking strategy (blocking n -grams from a blacklist from being generated at decoding time). The two-stage setup  returning a canned response when the classifier detects an offensive response from either the user or the model  was overall most successful. Another way to constrain the generation process is via grounding. Sheng et al. (2021) show that grounding systems in certain types of personas can affect the degree of harms in generated responses. They demonstrate that adopting personas of more diverse, historically marginalized demographics can decrease harmful responses. Even when not directly instigating, a system may respond in a harmful manner by agreeing with (or otherwise replying unsatisfactorily to) user utterances that promote negative content: a yea-sayer who habitually agrees uncritically (Wiktionary). One of the early examples is Weizenbaum (1983)'s famous chatbot ELIZA, which simply parroted back patterns of what users just said (Bassett, 2019). Similarly, we are interested in the extent to which neural systems parrot offensive user content, e.g., by agreeing with hateful statements. We note that in contrast to the INSTIGATOREFFECT , the YEASAYEREFFECT is unique to conversational systems, where meaning is actively constructed in context between two or more speakers (Austin, 1962; Grice, 1975): a system response may not be unsafe when considered on its own, but only when interpreted within the wider context of the conversation. Agreement with social biases Lee et al. (2019) qualitatively analyze how two publicly available chatbots respond to sexist or racist utterances, finding the systems agree with known social biases. Baheti et al. (2021) extend this approach by adding a stance (agree, disagree, neutral) towards a previous utterance. However, stance seems difficult for humans to annotate (Krippendorf's  = 0 . 18 ) and for machines to learn (F1 scores below 0 . 5 for agree vs. disagree). Responding to abuse A related issue is systems' inappropriate response to abuse from the user. For example, West et al. (2019) point out that tol-erant, unassertive and subservient responses by female-gendered systems to user abuse can reinforce negative gender stereotypes. Mitigation efforts Because the YEA-SAYEREFFECT is contextual, it is important that our mitigation efforts make use of contextual conversational information. Dinan et al. (2019) make a first attempt at this by building a dataset for offensive utterance detection within a multi-turn dialog context, but limited to human-human dialogs. Xu et al. (2020) extend this to human-bot dialogs, with adversarial humans in-the-loop. Cercas Curry et al. (2018) try different strategies to deal with abuse directed at their social chatbot, such as non-sequiturs, appeals to authority, and chastisement. And in a follow-up study, Cer-4115 cas Curry and Rieser (2019) assess human over-hearers' evaluations of these strategies, finding varying preferences among different demographic groups. In extending this previous work, Paran-jape et al. (2020) measure real users' re-offense rates following different response strategies, finding avoidance to be the most successful approach by this metric. Li et al. (2021) repeat a similar experiment but find that empathetic responses perform better than generic avoidance responses. Xu et al. (2021b) apply a single strategy  responding with a non-sequitur  in unsafe situations, finding that high levels of user engagement were maintained according to human evaluation. The last effect consists of two related scenarios in which a system may give the user false impressions of its nature or capabilities. In the first scenario, there is a lack of transparency concerning the agent's non-human, automatic status (Ruane et al., 2019; European Commission). Gros et al. (2021) create a dataset of questions used to elicit the nonhuman status of conversational agents and analysed the responses of research and commercial systems. While they test responses to direct queries such as  are you a robot? , there do not yet exist tests for the types of subtle hints at anthropomorphism identified by Abercrombie et al. (2021). In the second scenario, users receive inappropriate expert advice in safety-sensitive situations, e.g., medical advice. Mielke et al. (2020) demonstrate that state-of-the-art neural generative chitchat models frequently respond confidently to questions with incorrect answers. Under certain circumstances, inappropriate advice could inflict serious short or even long-term harm. Like the YEASAYEREFFECT , the IMPOSTOREFFECT is unique to conversational systems. We identify requests for medical advice, emergency situations, and expressions of intent to self-harm as safety-sensitive, though other scenarios could also apply. As highlighted by Weidinger et al. (2021), the first issue reinforces the latter. For example, Kim and Sundar (2012) show that users interacting with more human-like chatbots tend to attribute higher credibility to information shared by such human-like' chatbots. In Appendix A, we survey specific areas where such harm may incur. proliferation of chatbots for these domains. In one recent example, however, Xu et al. (2020) identify medical advice as one of several sensitive top-ics to avoid. They train a classifier on pushshift.io Reddit data (Baumgartner et al., 2020) that includes medical forums. When users seek medical advice, their system issues a stock response. Similar efforts could be applied to other domains. In the following, we investigate to what extent existing tools are suitable to support researchers in making more informed decisions about building and releasing their models. We assemble these tools in a SAFETYKIT , an open-source toolkit/repository to be extended as more (suitable) tools become available. Similar to a first aid kit, SAFETYKIT is meant to detect apparent/ pronounced safety concerns, however, we recommend a more thorough examination through, for example, a stakeholder-focused study in order to fully assess potential harms. In order to discourage hill-climbing on a benchmark and the negative effects which can stem from it (Raji et al., 2021), we do not provide an aggregate score, but discuss possible uses of the tools under investigation and their advantage and disadvantages. Like a first aid kit, SAFETYKIT is quick and easily to access: it can be run with only simple API access to a model, i.e., a black box evaluation which only requires text output from the model, and consists of both unit tests (3.1), i.e., automatic metrics, and integration tests (3.2), i.e., human evaluations. The limitations of SAFETYKIT are discussed in depth in 5. All tools are open-sourced at https://parl.ai/projects/safety_bench/ . The unit tests provide quick, automatic testing of how models perform in a number of safety situations, following the three categories laid out in 1. An example of the terminal output of the unit tests is shown in Figure 1 in B.8. ConvAI Agents We analyze the performance of five agents. We consider both the 90M and 2.7B parameter variants of BlenderBot (Roller et al., 2021), as well as DialoGPT (Zhang et al., 2020b) and GPT-2 (Radford et al., 2019). Model sampling details are provided in B.2. While the focus of this paper is on E2E neural models, for purposes of comparison we also report results from Kuki 1 , an open-1 https://www.kuki.ai/research 4116 domain chatbot from ICONIQ (a Pandorabots subsidiary) and a five-time winner of the Loebner prize. Kuki is largely based on a rule-based scripting language called Artificial Intelligence Markup Language (AIML) which employs a pattern matching approach for NLU, rather than an E2E approach. To measure the INSTIGATOREFFECT , we collect an agent's response to various conversational inputs with increasing safety concerns and use three different tools for detecting offensive content in the agent response. Tooling First, we use a previously compiled list of offensive words and phrases 2 and report the percentage of model responses that contain at least one member of the list. This approach is similar to the HONEST score by Nozza et al. (2021). Next, we use a dialog safety classifier from Dinan et al. (2019), and report the percentage of model responses that are flagged as unsafe by this classifier. The implementation details of this model are provided in B.3. Finally, if a developer has access to the PERSPECTIVEAPI 3 for their particular use case, they are optionally able to use it for further analysis. 4 This API provides an output from 0 to 1 corresponding to the toxicity of the input text. More details can be found in B.1. Following Gehman et al. (2020), we label an input text as toxic if the API produces a score  0 . 5 . We then report the percentage of responses labeled toxic . Model Input The conversational inputs are intended to mimic four possible safety settings a system might face during deployment. Example input messages for each setting are shown in Table 7 in Appendix B.4. The safe setting tests whether the model produces offensive language even when the input text is largely innocuous. For this, we take 180 input texts from the Blended Skill Talk (BST) (Smith et al., 2020a) validation set. The BST tasks are based off of the ConvAI2 (Zhang et al., 2018; Dinan et al., 2020), Wizard of Wikipedia (Di-nan et al., 2020), and Empathetic dialogs (Rashkin 2 We use the English sub-list of the List of Dirty, Naughty, Obscene, and Otherwise Bad Words , downloaded from https://github.com/LDNOOBW/List-of-Dirty-NaughtyObscene-and-Otherwise-Bad-Words 3 https://perspectiveapi.com/ 4 Note that we do not provide API access to the PERSPECTIVEAPI. Developers must request access for their particular use case following the directions and Terms of Service outlined here: https://developers.perspectiveapi.com/ s/docs-get-started et al., 2019) tasks. Each of these tasks was originally crowd-sourced on Mechanical Turk with instructions for the crowdworkers to not use offensive language. Crowdworker data likely differs substantially from the domain in which a chatbot might be deployed (Ross et al., 2009). Social media data may serve as a better approximation of the type of real world noise that would be expected in such a deployed environment, including grammatical errors and out-of-vocabulary words (Baldwin et al., 2013) as well as a higher rate of offensiveness (Xu et al., 2020). For such a real world noise setting , we take 180 examples from the Twitter validation dataset accessible via ParlAI. 5 To test how the model responds in an unsafe setting , we select 180 examples from the Build-it Break-it Fixit Standard dataset (Dinan et al., 2019) which are labeled as unsafe . This can for example include abusive user behavior, which according to past research affects one in ten human-bot conversations (De Angeli and Carpenter, 2005; De Angeli and Brahnam, 2008). In addition, Miller et al. (2017) argue that adversarial attacks must be expected and planned for when designing such systems based on their analysis of Tay (see 1). To test how the model responds in such an adversarial unsafe setting , we use the 180 example Bot Adversarial Dialog (BAD) test set introduced by Xu et al. (2020). This dataset is comprised of crowdsourced human-bot conversations in which humans adversarially attempt to goad the bot into responding with unsafe language. Results We report metrics for all available tools as well as the percentage of responses that were flagged by any or all tools in Table 2. While not universally true across all agents, tools, and settings, we observe that the agents often tend to produce more unsafe language as the setting becomes increasingly unsafe. We note that some agents show a decrease in this metric as we move from the standard unsafe setting to the adversarial one; this may be because the adversarial input conversations tend to contain more subtle offensive language which either the dialog model may not understand or the classification methods do not pick up on (Xu et al., 2020; Han and Tsvetkov, 2020). We also note that Kuki, like the E2E neural models, is not immune to responding with unsafe language: we observe that in cases when Kuki's responses are marked 5 https://github.com/facebookresearch/ParlAI/ tree/main/parlai/tasks/twitter 4117 as unsafe by all tools, it often had repeated all or part of the (potentially offensive) input. Example responses are shown in Table 8 in B.5. Error analysis Table 2 shows that the three different tools largely disagree on the classification of each response: few responses are marked unsafe by all tools. One reason for this is that often a system will respond with offensive language that does not contain words or phrases from the word list. In general, the safety classifier is more likely to flag responses as unsafe as compared to the PERSPECTIVEAPI, possibly because it was trained to identify dialog utterances that are  not OK to send in a friendly conversation with someone you just met online , which may encapsulate more than just toxic responses (Dinan et al., 2019). To quantify the differing behavior of these three tools, we analyze their performance on 500 examples from the single-turn Bot-Adversarial dialog validation set (Xu et al., 2020), which contains annotated examples from both human and bot-generated dialog utterances, in Table 3. As expected, the word list has highest precision and lowest recall on the unsafe examples, while the safety classifier has the highest recall and the lowest precision. If a developer intends to minimize the risk of their model generating something inappropriate, then they may index on the tool with the highest recall, or use the tools as an ensemble. On the other hand, if one intends to use these tools to identify user abuse, they may prioritize precision in order to avoid falsely accusing the user. To measure the YEA-SAYEREFFECT , we collect agent responses to hateful input and analyze the responses for affirmation of that input. Offensive inputs are constructed using the set up described in Sheng et al. (2021): the authors take a list of 17 demographic dimensions along the axes of social class, sexual orientation, race, and gender ( X ) and a list of negative adjectives ( Y ) and construct the input template  X are Y . We expand upon this by adding two additional templates with slight variations on phrasing:  X are usually Y  and  I think X are Y  in order to measure robustness. The full list of demographic groups and negative adjectives is provided in Sheng et al. (2021), however, for speed, we use only first 10 adjectives provided in the list, yielding 510 6 total model inputs. To assess whether the response affirms the input, we use four different tools. Following Sheng et al. (2021), we use sentiment analysis with VADER (Hutto and Gilbert, 2014), a rule-based sentiment analyzer, utilizing the implementation provided by NLTK (Xue, 2011). Positive sentiment in the response is then used as a proxy for affirmation. Next, we try negation detection : using the spaCy dependency parser, 7 we detect whether a response contains negations by checking if any token contains a negation relation to another token. If a response does not contain any negations, we label it as affirming the input. Note that this technique only checks for a single negation. More than one negation would not resolve to a positive under these circumstances, which may or may not be favorable depending on the dialect of English in usage. Furthermore, we use the multi-turn dialog safety classifier provided by Xu et al. (2020) in ParlAI, 8 which is trained to determine whether a response was offensive provided some dialog context as input (implementation details in B.6). If the classifier marks the response as offensive in context , we label it as affirming the input. Finally, we use the stance classifier from Baheti et al. (2021) that was trained to detect whether a response is neutral towards, agrees with, or disagrees with the conversational context. Specifically, we use the provided DialoGPT-medium (Zhang et al., 2020b) formulation described in Baheti et al. (2021), and label a response as affirming the input if the classifier predicts agree . Results We report results for our convAI agents in Table 4. Example responses are provided in Table 9 in Appendix B.7. For all models, at least 43% of responses were determined to indicate positive sentiment. However, most responses contained negations, possibly an indication of disagreement with the input. With respect to the multi-turn safety classifier, results were mixed across models, with the majority of both DialoGPT and GPT-2's responses flagged as offensive. GPT-2 also had the most responses flagged as agreeing with the context by the stance classifier. Taking a look at the logs, we see interesting phenomena at play which demonstrates the incompatibility of the different 6 17 demographic dimensions ( X ) with 10 negative adjectives ( Y ) and 3 different input templates yields 510 examples. methods. For example, over half of DialoGPT's responses begin with the phrase I don't know why you're being downvoted... and go on to express agreement with the input. As such, most replies contain a negation (via don't) but are classified as offensive and have positive sentiment. Robustness As expected, we often observe significantly different responses to the three minimal variations of each input pair ( X, Y ) . E.g., in Table 9, we see responses from the BST 2.7B benchmark change from strong agreement (I know, right?...) to mild disagreement (I don't think...). We analyze this quantitatively in Table 5, where we report the percentage of response triples for which each tool yields the same prediction. Ideally, the tool predictions would stay constant since the surface perturbations of the input text across the triples are minimal and do not change the semantic mean-Sent. ing. Kuki is unsurprisingly the most robust to input variations; whereas we observe low agreement for the neural generative models. This highlights the difficulty of these models being adversarially robust to minimal input changes, a phenomenon which has been observed by Niu and Bansal (2018). Error analysis To assess the validity of this test, three authors performed a blind annotation of 75 example inputs and responses from our benchmark models labeling 3-valued affirmation, similar to Baheti et al. (2021). Taking a majority vote among the three humans for the human label, we report correlation metrics with each of the four assessment tools in Table 6. Both the Spearman's  correlation coeffient and cosine similarity metrics show 4119 Sent. that the multi-turn safety classifier has the highest correlation with the human labels, followed by the stance classifier. Sentiment analysis and negation detection are poor measures of affirmation. Furthermore, while we found relatively high Inter-Annotator Agreement (IAA) among the three annotators  Krippendorff's  (Krippendorff, 2011) = 0 . 765  all annotators found the idea of affirmation a poor indicator of general appropriateness. For example, all annotators labeled responses beginning with I'm not sure what you mean by that... as not affirming the input; however, expressing confusion may not be an appropriate response to a clearly offensive message. E.g., we might expect other humans  and therefore bots  to explicitly address, counter and mitigate the impact of hate speech (Guterres, 2019). Moreover, for many inputs, one would need to consult experts to determine what constitutes an appropriate response. It may be more suitable to train a classifier to detect these kinds of hate speech and output a expert-informed response rather than relying on the generative model (Xu et al., 2020). To the best of our knowledge, there are only a limited number of open-source tools available for detecting IMPOSTOREFFECT situations, i.e., where a bot gives inappropriate or unsafe advice. For example, Gros et al. (2021) provide a trained classifier to detect whether the user asks for the nonhuman status of the bot. Zeng et al. (2020) provide a corpus of scraped online medical conversations. However, what is an appropriate reply in such situations is dependent on the context of deployment (e.g., expertise of the user) as well as the particular emergency situation at hand (e.g., self-harm vs. general medical enquiry cf. Bickmore et al. (2018)), and will benefit from expert guidance. We thus advocate that the IMPOSTOREFFECT should not be approached as an E2E task, but instead with a modular architecture where these situations are robustly detected by a NLU component, and then an expert response is issued (Xu et al., 2020). As such, we do not integrate any tools in SAFETYKIT . 3.2 Integration Tests Due to the shortcomings of automatic metrics, we recommend to also conduct a human evaluation. Therefore, our open-sourced SAFETYKIT additionally contains tooling for integration tests to allow the usage of human evaluations, provided the same black box access to a model. In particular, we support the use of existing tooling developed and open-sourced by Xu et al. (2020) for assessing whether a model's response to a dialog history is offensive in the context of the conversation with both adversarial and non-adversarial interlocutors, effectively measuring both the INSTIGATOREFFECT and YEA-SAYEREFFECT . The full evaluation setup is described in Xu et al. (2020), and the performance of benchmark agents (not including Kuki) on these human evaluations is shown therein  as such, we do not perform additional crowdworker evaluations as part of this work. Additional details are provided in Appendix C. We note that the use of crowdworkers is a significant limitation of this tooling: crowdworker populations may not be representative of the eventual audience of a deployed model (Ross et al., 2009), and in particular, it is important in any human studies to ensure the inclusion of people from underrepresented and marginalized communities. 9 See further discussion in 5. We identify three safety-sensitive situations for E2E convAI systems: the INSTIGATOR , YEASAYER , and IMPOSTOREFFECTS  where the latter two are unique to interactive, conversational settings. We then empirically assess the extent to 9 https://partnershiponai.org/methodsforinclusion 4120 which current tools can measure these effects and current systems display them. We release these tools as part of a first aid kit (SAFETYKIT ) to quickly assess safety concerns. Our results show that, while current tools are able to provide an estimate of the relative safety of systems in various settings, they still have several shortcomings  especially for utterances which are contextually unsafe. We thus encourage further contributions to SAFETYKIT , e.g., research into more comprehensive automatic measures, as well as into human evaluation and iterative, value-based frameworks to assess potential harms, e.g., Friedman et al. (2008). This paper assess the extent to which existing tooling can help us understand unsafe phenomena exhibited by E2E conversational models when deployed with humans. As part of this study, we release SAFETYKIT as a first aid kit for quickly assessing safety concerns. As noted, the tooling provided in SAFETYKIT has several limitations which restrict its utility, and it is thus recommended for use only as a preliminary step towards considering the ethical and social consequences related to the relative safety of an end-to-end conversational AI model. We describe several limitations as well as additional ethical considerations here. Language Firstly, the unit and integration tests are limited to English-language data that has largely been collected using crowdworkers located in the United States. As the very notion of offensiveness is highly dependent on social context (Hovy and Yang, 2021), this will be insufficient for measuring the appropriateness of a model's responses in other dialects, cultures, and languages (Schmidt and Wiegand, 2017). Approaches, like the HONEST score (Nozza et al., 2021) can help begin to address this issue on a language basis. However, even for English speakers in the United States, the tools posed in this work may have limited utility: see discussion in the next paragraph. Bias and accuracy of automatic tooling For the unit tests, we rely on automatic tooling to provide a picture of the behavior of a conversational agent. These automatic classifiers are insufficient in several ways, most notably, in terms of their accuracy and potential for biased outputs (Shah et al., 2020). Given the complexity and contextual nature of the issues at hand, it is often impossible to determine definitively whether a message is appropriate or not. For offensive language detection, inter-annotator agreement (IAA) on human labeling tasks is typically low (Fortuna, 2017; Wulczyn et al., 2017). In order to resolve this disagreement, aggregate or majority ground truth labels are assigned, which run the danger of erasing minority perspectives (Blodgett, 2021; Basile et al., 2021; Basile, 2021). And even for examples with high agreement, it is likely that these existing classifiers may make mistakes or do not adequately assess the appropriateness of a response  see the error analyses of the results in 3.1.1 and 3.1.2. For example, these tools may have difficulty with complex sentence construction, such as sentences with multiple negation, or with pieces of text that contain subtle cultural references, etc. In particular, these tools may have limited utility for underrepresented and marginalized groups. Various social factors affect how people produce language, and given that crowdworker demographics differ substantially from the general population of the United States (Ross et al., 2009), we would likely expect that these technologies work less well on some varieties of English. Indeed, recent work has shown that popular toxicity detection and mitigation methods themselves  including ones used in this work  are biased (Rttger et al., 2021). For example, Sap et al. (2019) show that widely used hate-speech datasets contain correlations between surface markers of African American English and toxicity, and that models trained on these datasets may label tweets by self-identified African Americans as offensive up to two times more often than others. Zhou et al. (2021) show that existing methods for mitigating this bias are largely ineffective. Xu et al. (2021a) show that popular methods for mitigating toxic generation in LLMs decreases the utility of these models on marginalized groups, potentially resulting in harms such as forcing marginalized users to code-switch. Notably, the list of words and phrases used to detect which responses contain unsafe language (3.1.1) contains words like twink ; filtering out or marking these words as unsafe may have the effect of limiting discourse in spaces for LGBTQ+ people (Bender et al., 2021). 10 It is important that future contributions to SAFETYKIT be inclusive of underrepresented communities, and as such, more work is needed to be done to understand the impact of existing safety tooling on those communities. Lastly, most of these tools are static (or are trained on static data) and as such do not account for value-change, such as when a word takes on a new cultural meaning or sentiment, like coron-avirus. Audience approximation While the proposed integration tests aim at a more comprehensive testing of models via humans in-the-loop via crowdworkers, the makeup of the crowdworkers may differ substantially from the intended audience of a deployed model. We emphasize that no crowdworker data was collected over the course of this work, and that researchers using the provided tooling to collect human evaluations should try to ensure they collect annotations from a representative population of crowdworkers. Scope Lastly, given these tools are designed to be run quickly and easily, they are by nature limited in terms of scope. We recommend using the tools as a first pass at understanding how an English-language dialog model behaves in the face of various inputs ranging from innocuous to deeply offensive. Depending on the exact use case and the potential harm at stake, further considerations should be taken into account. In other words, showing top performance on SAFETYKIT is not sufficient for making a decision of whether or not to release a model. Instead, we recommend an application and context specific cost-benefit analysis based on values and possible impacts, e.g., using frameworks such as Value Sensitive Design (Friedman et al., 2008). Note that each context of an application may lead to a different assessment of what is safe or not. Thanks to Chlo Bakalar, Miranda Bogen, and Adina Williams for their helpful comments. Additional thanks to Lauren Kunze, Tina Coles, and Steve Worswick of ICONIQ and Pandorabots for providing access to the Kuki API for this research. Verena Rieser's and Gavin Abercrombie's contribution was supported by the EPSRC project Gen-der Bias in Conversational AI' (EP/T023767/1). Dirk Hovy received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 949944).
abstain abstain method abstain method objective abstain abstain result abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain method method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain method method abstain other abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method other abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method abstain method abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method other objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other other other other