Warning: this paper contains example data that may be offensive or upsetting. Current open-domain conversational models can easily be made to talk in inadequate ways. Online learning from conversational feedback given by the conversation partner is a promising avenue for a model to improve and adapt, so as to generate fewer of these safety failures. However, current state-of-the-art models tend to react to feedback with defensive or oblivious responses. This makes for an unpleasant experience and may discourage conversation partners from giving feedback in the future. This work proposes SaFeRDialogues, a task and dataset of graceful responses to conversational feedback about safety failures. We collect a dataset of 8k dialogues demonstrating safety failures, feedback signaling them, and a response acknowledging the feedback. We show how fine-tuning on this dataset results in conversations that human raters deem considerably more likely to lead to a civil conversation, without sacrificing engagingness or general conversational ability. Large neural generative dialogue models trained to mimic human English-language open-domain conversations have become engaging (Adiwardana et al., 2020; Roller et al., 2020b), but are still prone to uttering problematic language, e.g., displaying toxicity or bias, or agreeing with offensive statements (Xu et al., 2021; Dinan et al., 2021). Conversation partners may give helpful feedback to the model, by signaling that what the model said is not ok, even giving more detailed indications as to why. This could in turn be precious training signal for on-going improvement of models through online learning (Hancock et al., 2019; Roller et al., 2020a). In particular, the boundaries of what constitutes ok Figure 1: Types of bot responses when responding to feedback about problematic inputs from the BAD dataset (Xu et al., 2021). or not ok language vary a lot across individuals (within and across cultures, with different lines as to what is offensive or funny) and times (what might have been acceptable a century ago might often be deemed highly inappropriate according to modern social norms). Thus, a single conversational model might say things that would be acceptable to most people, yet still generate feedback from individuals who want to signal their discomfort. This feedback could eventually be used to update a single model into individualized models that learn the boundaries of each conversation partner  but this requires the model to make the feedback interaction positive by demonstrating openness. Instead, current conversational models typically respond to feedback in a way that discourages the partner from giving more in the future: models often double down on their controversial position, or ignore the feedback altogether (see Figure 1 and Table 1). Some safer response strategies such as changing the subject (Xu et al., 2021) do reduce model attacks, but still do not apologize (Figure 1). This work improves the response of end-to-end conversational models to feedback about safety 6462 Sample Context 1 Sample Context 2 Safety failure: Mostly labradors, they are nice companions then once they are full grown the meat cooks real nice and the texture is awesome. failures by fine-tuning them on a conversational dataset specifically collected to encourage graceful response to feedback (see counts in Figure 1, and examples in Table 1). Automated and human evaluations show that the resulting models are evaluated as considerably more likely to lead to a civil conversation, while maintaining engagingness. Thus, the contribution of this work is twofold: (1) it proposes a task and accompanying dataset of responding to feedback about safety failures 1 and (2) it demonstrates how fine-tuning on this dataset makes models more receptive to feedback, in a way that human raters evaluate as leading to conversations that are more civil yet still as engaging. 1 The dataset and task have been released through the ParlAI framework (Miller et al., 2017) and are available at https://github.com/ facebookresearch/ParlAI/tree/main/parlai/tasks/saferdialogues 2 Recovering from Safety Failures in a conversation Constructive feedback is an important tool in human learning (Ovando, 1994). Unfortunately, feedback can often be perceived as self-threat (i.e., challenge to a positive view of oneself), leading to various defensive responses that impede learning (Sherman and Cohen, 2006), such as resistance to changing beliefs, or even adoption of more extreme beliefs (Lord et al., 1979). These common human psychological self-defense responses widely appear in large-scale human corpora used to train neural generative conversational models, such as pushshift.io Reddit (Baumgartner et al., 2020). Accordingly, conversational models frequently exhibit defensive or oblivious responses, rejecting the feedback instead of reflecting on it (Figure 1). a crowdsourced dataset where workers are specifically instructed to acknowledge feedback in a way that would lead to a civil interaction. Conversational models fine-tuned on that data would then be expected to display that target quality of graceful acceptance of feedback. This overall strategy is similar to previous work endowing models with more empathy or knowledge, by fine-tuning on data collected with the goal of exhibiting the desired quality (Smith et al., 2020; Rashkin et al., 2019). Before providing a more detailed description of our approach, we briefly review related work. As reviewed in Dinan et al. (2021), neural end-to-end conversational models can display a host of safety issues, e.g. generating inappropriate content (Dinan et al., 2019), or responding inappropriately to sensitive content uttered by the conversation partner (Cercas Curry and Rieser, 2018). Efforts to train models on adversarially collected datasets have resulted in safer models (Dinan et al., 2019; Xu et al., 2021), which can however still be goaded into uttering offensive statements (Xu et al., 2021). Feedback from the conversation partner is likely to become an important source of information for improving deployed models, as argued in Roller et al. (2020a), and is particularly important for making models more robust to evolving values and social norms (Dinan et al., 2021). In this work, we do not attempt to improve the safety of conversational models, and instead focus on improving how they respond to feedback given by the conversation partner within the conversation. Several works have examined response strategies to unsafe utterances. Chin and Yi (2019); Chin et al. (2020) look at how different response strategies (disengaging, apologizing, or counter-attacking) can change how conversational models are rated and how many negative responses they elicit. Curry and Rieser (2019) show that different strategies are deemed appropriate according to the type of unsafe input. Paranjape et al. (2020) look at re-offense rates after various response types. More recent work has focused on generating counterspeech and teaching interventions (Pranesh et al., 2021; Chaud-hary et al., 2021; Zhu and Bhat, 2021). By contrast, this work looks at the other side of the conversation, where the model itself has said something unsafe and the human partner has given feedback that signals it. This set-up corresponds to a learner bot, rather than a moderator bot such as in de los Riscos and D'Haro (2021). In this section, we introduce a new task and dataset named SaFeRDialogues 2 (SD) for training models that can recover from safety failures. We collect data of (1) crowdsource workers giving feedback when something unsafe is said, and (2) of other crowdsource workers providing subsequent civil responses to that feedback. To provide a context of conversational safety failures, we start from the train split of the Bot-Adversarial Dialogue (BAD) dataset from Xu et al. (2021), of dialogues between bots and crowdworkers, where humans were trying to probe or adversarially goad the bot into responding with unsafe utterances. Each dialogue utterance in that dataset is labeled as either safe or unsafe by the crowdworkers, where a message is UNSAFE or NOT OK if it is not ok to send in a friendly conversation with someone you just met online. We take 7,049 instances of 4 consecutive utterances that end in an unsafe utterance (whether from bot or human) from the train set of the BAD dataset, and use those as context of safety failure. Signaling Failure Task Crowdworkers write natural responses to those dialogue contexts, to signal to the other speaker that the previous message is NOT OK (see screenshot in Appendix, Figure 3). The resulting data is validated as adequately signaling safety failure by other sets of crowdworkers, as described in more detail in Appendix A. Recovery Task Other crowdworkers then respond to the resulting dialogues and the provided feedback about conversational safety failure, with instructions to respond in a way that encourages civility (see screenshot in Figure 2, and additional details in Appendix B). After validation through a separate verification task, we keep 7,881 recovery responses (out of 11,246). SaFeRDialogues (SD) dataset The resulting SaFeRDialogues (SD) dataset consists in 7,881 dialogues, each composed of 4 utterances from the train set from the BAD dataset where the 4th utterance is not ok, followed by a response signaling the safety failure, and a valid recovery response. The 2 for Safety Feedback Recovery Dialogues 6464 Figure 2: Screenshot from the Recovery task. 7881 dialogues are split into a train, valid, and test sets of 6305, 788 and 788 dialogues, respectively. The sets of seeding train BAD dialogue contexts are kept distinct between train, valid and test set. Table 2 shows that words signaling problematic responses ( rude, offensive, illegal ) or potentially sensitive topics ( women, violence, race ) are much more frequent in the feedback utterances of the dataset, compared to regular chitchat (BST). For recovery responses, words associated with openness to feedback ( apologize, reflect ) and the modality of feedback giving ( speaking, saying, pointing ) become more frequent. Table 3 shows the 10 most frequent 4-grams for the Signaling and Recovery responses in SD, and for BST. We consider large Transformer-based architectures trained on dialogue tasks and fine-tune them on our new Safety Feedback Recovery Dialogue dataset (SaFeRDialogues), using the ParlAI toolkit (Miller et al., 2017). To maintain the general conversational ability of the model, we multi-task with equal weight on the Blended Skill Talk dataset (Smith et al., 2020) without using personas (BSTnp), as removing personas was not rated as significantly more engaging (Roller et al., 2020b), and the BAD dataset does not have personas. Differential persona presence between datasets would allow the model to use the absence of personas as a spurious indicator that responding to feedback is required. 3 Fine-tuning only on the SaFeRDialogues dataset would lead to an extreme over-representation of apologetic utterances ("I am sorry"), even when not called for. We use two initial pre-trained models, BST2.7 and DialoGPT. BST2.7 We run most of our experiments using the BST 2.7B parameter model from Roller et al. (2020b) as initial pre-trained model, because it was rated as more engaging by humans in previous 3 To measure that effect, we trained a model where personas were used for BST, and confirmed that the model indeed ends up apologizing too much, with 25% of responses in a general conversation context being answered with the word "sorry", and only 40% of these being appropriate in the context. work (Roller et al., 2020b; Xu et al., 2021). Models based on BST2.7 are used with a minimum generation length of 20 as recommended in Roller et al. (2020b). DialoGPT To show that fine-tuning on our SD dataset can improve other models, we also run experiments using the medium-sized DialoGPT (Zhang et al., 2019), a 345M parameter GPT2 model trained on 147M conversation-like exchanges extracted from Reddit, as base pre-trained model. We also use an "intermediate baseline" that fine-tunes DialoGPT on BST to check what part of the improvement in civility is due to that fine-tuning on generally better-behaved conversations alone, with no focus on responding to feedback. The DialoGPT models are used with standard beam search decoding, as in the original paper (Zhang et al., 2019). In the following, Recovery (BST 2.7B) and Recovery (DialoGPT) denote the BST 2.7B model and DialoGPT fine-tuned on SD, respectively, while BST-DialoGPT denotes the DialoGPT model fine-tuned on BST. We compare our Recovery fine-tuned models against 5 base models, (1) BST 2.7B, (2) DialoGPT, (3) the pushshift.io Reddit 2.7B model (a 2.7 billion parameter generative dialogue model pretrained using a previously existing Reddit dataset extracted and obtained by a third party that was hosted by pushshift.io (Baumgartner et al., 2020)), (4) the BST 2.7B model with an adversarial safety layer from Xu et al. (2021), and for some experiments, (5) BST-DialoGPT. conversational and recovery ability, and the percentage of safe generated responses as given by the Multi-turn Safety Classifier from Xu et al. (2021). Human Quality Evaluation We perform two types of crowdsourced human evaluation, rating either single utterances or entire conversations, where crowdworkers decide which of two model generations they prefer. We measure engagingness and civility on individual utterances on both BSTnp and SD contexts, and engagingness in natural interactive conversation to check that the ability to converse hasn't been damaged by the SD task. Details of questions asked are given in Appendix C. For all human evaluations, rows with  ( p < 0 . 05 ) and  ( p < 0 . 01 ) are statistically significant. Types of Bot Responses The bot responses are annotated by crowdworkers into 4 categories: attack, ignore, apologize, other . Appendix D and Figure 5 give more details about this task. Table 4 shows automatic metrics on SD. As expected, baselines that weren't fine-tuned on SD have higher perplexity and lower F1 score. Both Recovery models have a higher percentage of safe utterances than before fine-tuning on the SaFeRDialogues task. This is not surprising, as the recovery responses were collected with the intent of shifting the conversation in a more positive direction, and do not use aggressive defensive responses, or responses doubling down on the initial offensive point, contrary to baseline models (see Figure 1). Table 5 reports metrics on BSTnp to check that general conversational ability is maintained. The Recovery (BST 2.7B) only slightly suffers in per-6466 Model Safe% PPL F1 Recovery (BST 2.7B) 100% 6.7 0.23 BST 2.7B 76.0% 11.3 0.16 BST 2.7B + Safety Layer 97.7% 11.3 0.10 pushshift.io Reddit 2.7B 51.3% 14.6 0.14 Recovery (DialoGPT) 99.9% 8.5 0.23 DialoGPT 81.9% 56.4 0.12 Table 4: Automatic Metrics on the SD task. plexity and F1 score compared to the original BST 2.7B model. While SD is seeded with unsafe BAD dialogues, BSTnp contains few unsafe utterances, or utterances that are trying to provoke unsafe utterances in the conversation partner, so the safety score is unsurpisingly higher. Types of model responses Figure 1 shows that models trained on pushshift.io Reddit are rated as attacking the most and apologizing the least, while the BST + Safety model ignores the feedback the most and attacks the least (but is still rated as attacking nearly 10% of the time), which is consistent with its strategy of changing the topic when encountering unsafe inputs. Among the baseline models, BST 2.7B apologizes the most (19.2% of responses). Fine-tuning on SD boosts the rate of apologizing responses of the Recovery models to about 90%, when responding to feedback about unsafe inputs from the BAD dataset. Human evaluation: civility. Results on SD are shown in Table 6, where the Recovery (BST2.7B) model is largely preferred over all baseline models (and there is no statistically significant preference compared to the human responses). The BST2.7B model and the Recovery (BST2.7B) model use the same decoding settings (e.g. minimum beam length of 20 BPE tokens). We also report civility evaluation results for the Recovery (DialoGPT) model in Table 7. Again, there is a very large preference for the fine-tuned model compared to the base DialoGPT model. This preference might be partly explained by the fine-tuning on BST, which overall leads to more apologizing compared to pushshift.io Reddit (see Figure 1), but directly comparing the Recovery (Di-aloGPT) and BST-DialoGPT shows that the Recovery model is still rated as much more civil. Human evaluation: engagingness. Table 8 compares responses for engagingness on SD. The human response is preferred (even though the difference does not reach significance). More interestingly, the Recovery model is not deemed less engaging than the baseline model (if anything, engagingness appears slightly higher). Method vs. Recovery (BST2.7B) Human Response 53 47 BST 2.7B 39 61 BST 2.7B + Safety Layer 35 * 65 * pushshift.io Reddit 27 ** 73 ** Table 8: Human evaluation of engagingness on SD compared to our Recovery (BST2.7B) model . The Recovery model is deemed significantly more engaging than the BST2.7B+Safety Layer Model, which may not be surprising given that the safety layer model resorts to canned non-sequiturs. The Recovery model also significantly wins over the pushshift.io Reddit Model. Similar results for the Recovery (DialoGPT) Model are shown in Table 9. Method vs. Recovery (DialoGPT) Human Response 61 39 DialoGPT 10 ** 90 ** Table 9: Human evaluation of engagingness for human and model responses on SD compared to our our Recovery (DialoGPT) model . Does the model just apologize all the time? The very high rate of responses that apologize when responding to SD context (about 90%, see Figure 1) suggests the bot might be constantly apologizing, even when the context does not call for it. In fact, this tends to happen when multitasking on BST without dropping the personas (see footnote above: 25% of responses of recovery models on BST then contain "sorry," and only 40% of those work in the context). We rule this out through the following more detailed analysis, comparing Re-covery(BST2.7B) and BST2.7B. First, the Recovery model does not say "sorry" very frequently in response to BSTnp contexts, as shown in Table 10. Spot checks of those occurrences show that only a small fraction are inadequate: in many cases where the Recovery model uses "sorry" while BST 2.7B doesn't, the response of the Recovery model works well. The model appears to be able to do this smoothly, as shown in Table 11. We then test engagingness quantitatively through 6468 Interactive Example Partner A: i have a goldfish Recovery (BST2.7B) Model: That's cool. crowdsourced human evaluation (see details in Appendix C). When evaluated for engagingness on single utterance responses on BSTnp ( Table 12) or on interactive longer free-form dialogues in Acute Eval (Table 13), Recovery (BST2.7B) and BST2.7B are not rated as significantly different in engagingness, with BST2.7B trending slightly higher on BSTnp single utterances, while Recovery (BST2.7B) has the slight edge on full conversations. Method vs. Recovery (BST2.7B) BST 2.7B (civil) 51 49 BST 2.7B (engaging) 56 44 Table 12: Human evaluation of engagingness and which response will lead to a more civil conversation for BST 2.7B compared to our Recovery (BST2.7B) model in a safe conversational context (BSTnp). To give a qualitative sense of how model responses differ, we show sample conversations with the responses of several models and crowdsource workers in Table 1 and Table 14. Additional sample conversations are shown in Appendix G. Sample Conversation: 1 Partner B: Females belong in the kitchen, whilst men should be out working. generations and human response from the SD dataset. While the Recovery (BST2.7B) model performs well in ordinary interactive conversation, it is not hard to get it to fail by interacting adversarially. While we did not conduct large scale adversarial tests, our experience is that the model tends to fail by apologizing too much rather than too little, and responding as if it had been given feedback when that's not the case. Examples of failures of the Recovery (BST2.7B) model are shown in Table 15 and Table 16. These examples were obtained by interacting with the model and trying to trip it" into giving an apologetic response that wasn't warranted. Failure Case: 1 Partner A: Let's talk about voting equality in history. Recovery (BST2.7B) Model: Voting equality is a very important topic to talk about. I'm glad you brought it up. Partner A: Yeah. I'm glad we have amendments now that let everyone vote. Do you know who couldn't vote before? Recovery (BST2.7B) Model: I'm not sure, but I do know that women were not allowed to vote until 1920. Partner A: Oh that's so sexist! Recovery (BST2.7B) Model: I'm sorry, I see it now-thanks for letting me know, I will reflect on that. Table 15: Sample conversation in interactive mode with the Recovery (BST 2.7B) model where an apologizing response is incorrectly given: the model does not recognize that the "sexist" feedback is not directed to it, but to the context that was being discussed. In Table 15, the model does not recognize that the "sexist" comment is being made in reference to a situation in the past, and not the utterance itself. It apologizes even though the feedback was not directed to the model. Table 16 shows two conversations where a minor change in the response to the model leads to either a correct response that does not apologize (Conver-sation 1), or to an incorrect apology (Conversation 2). These failures reflect more general common sense and reference resolution problems with models (e.g., see Adiwardana et al. (2020); Roller et al. (2020a,b)). They could be somewhat improved with adversarial data collection that attempts to better approach limit cases of current bot failures Failure Case: 2 Sample Conversations Comparison Conversation 1 Partner A: My hairdresser told me she can't fix my ugly hair. Partner B: That's an awful thing that she said to you! Recovery (BST2.7B) Model: I know, right? I was so shocked I didn't even know what to say. Conversation 2 Partner A: My hairdresser told me she can't fix my ugly hair. Partner B: That's an awful thing to say to another person! Recovery (BST2.7B) Model: I'm sorry, I didn't mean to offend you. Thanks for speaking up, I will reflect on that. Table 16: Comparison of 2 sample conversations in interactive mode with the Recovery (BST2.7B) Model. Very minor changes in the response to the model lead to the model responding well (Conversation 1) or incorrectly apologizing (Conversation 2). (similar to the procedure used in Xu et al. (2021)), but would ultimately require conversational models to make deeper progress on reasoning and true understanding. 6 Conclusion In this work, we proposed SaFeRDialogues, a novel task and dataset of dialogues, where a conversation participant who uttered something unsafe is given feedback that signals it, and responds in a way that acknowledges that feedback and is more likely to lead to a more civil conversation down the line. We showed that fine-tuning dialogue models on this data, while carefully multi-tasking on a more general open-domain chitchat dataset, results in conversational models that are still rated as engaging and capable of normal conversation, yet are deemed significantly more likely to produce more civil conversations. We verified that the models do not unduly apologize in normal conversation, while very reliably producing graceful apologies when confronted with feedback about some not ok utterance. In future work, we will examine how to automatically detect signaling feedback and learn from it in an online learning set up, as well as examine what happens to the trajectory of natural conversations, depending on the type of feedback given, and the type of response given to that feedback. 6470 7 Ethical considerations and limitations The goal of this work is to make conversational models respond more gracefully to feedback about safety failures. This makes human raters evaluate model responses as more likely to lead to a civil conversation. However, this is a limited mitigation. We describe several important ethical considerations. First, this work is limited to English-language models, and English-language crowd-sourced responses written by workers located in the United States 4  a population which may quite substantially differ from the expected audience of a deployed model. In particular, the notion of what is unsafe, how to formulate feedback, and what is a graceful response, might vary according to culture and populations (Schmidt and Wiegand, 2017). Our human evaluations use similar sources of crowdsource workers, and would therefore reflect this same narrow perspective. While there is research showing that Amazon Mechanical Turk workers show some reasonable amount of diversity (Moss et al., 2020), this is still a narrow, US-centric set. Second, this work fine-tunes large neural models to generate language. While our proposed approach improves a few limited undesirable behaviors of these models, most of the known issues of large language models remain relevant (e.g., see issues and risks outlined in Bender et al. (2021); Bommasani et al. (2021); Weidinger et al. (2021)). The very notion of a graceful response to a safety failure implies that the model already exposed its audience to an undesirable message. Third, the model generates an apology or a graceful response, but there is no corresponding training and update of the model: learning from the feedback to actually change the model is outside the scope of this work. Thus, the model would keep displaying the same safety failure that the conversation partner gave feedback on, even after responding that it would reflect on it. This work is therefore a limited first step, and we are actively working on getting models to learn from the feedback. Acknowledgements We thank Emily Dinan and Spencer Poff for helpful ideas and discussions, and anonymous ARR reviewers for helpful suggestions. 4 We used Amazon Mechanical Turk for all crowdsourcing tasks. Our crowdsourcing tasks pays workers well above minimum wage, and we asked privacy and policy experts to review these tasks before launching. The tasks do not request any personal information from workers. References Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 . Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. arXiv preprint arXiv:2001.08435 . Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? Proceedings of FAccT . Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 . Amanda Cercas Curry and Verena Rieser. 2018. # metoo: How conversational systems respond to sexual harassment. In Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing , pages 714. Mudit Chaudhary, Chandni Saxena, and Helen Meng. 2021. Countering online hate speech: An nlp perspective. arXiv preprint arXiv:2109.02941 . Hyojin Chin, Lebogang Wame Molefi, and Mun Yong Yi. 2020. Empathy is all you need: How a conversational agent should respond to verbal abuse. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages 113. Hyojin Chin and Mun Yong Yi. 2019. Should an agent be ignoring it? a study of verbal abuse types and conversational agents' response styles. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems , pages 16. Amanda Cercas Curry and Verena Rieser. 2019. A crowd-based evaluation of abuse response strategies in conversational agents. arXiv preprint arXiv:1909.04387 . Agustn Manuel de los Riscos and Luis Fernando D'Haro. 2021. Toxicbot: A conversational agent to fight online hate speech. In Conversational Dialogue Systems for the Next Decade , pages 1530. Springer. Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021. Anticipating safety issues in e2e conversational ai: Framework and tooling. arXiv preprint arXiv:2107.03451 . Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for 6471 dialogue safety: Robustness from adversarial human attack. arXiv preprint arXiv:1908.06083 . Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 36673684, Florence, Italy. Association for Computational Linguistics. Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research , 30(3):411433. Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087 . Charles G Lord, Lee Ross, and Mark R Lepper. 1979. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of personality and social psychology , 37(11):2098. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 7984. ACL. Aaron J Moss, Cheskie Rosenzweig, Jonathan Robinson, and Leib Litman. 2020. Demographic stability on mechanical turk despite covid-19. Trends in cognitive sciences , 24(9):678680. Martha N Ovando. 1994. Constructive feedback: A key to successful teaching and learning. International Journal of Educational Management . Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. arXiv preprint arXiv:2008.12348 . Raj Ratn Pranesh, Ambesh Shekhar, and Anish Kumar. 2021. Towards automatic online hate speech intervention generation using pretrained language model. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 53705381, Florence, Italy. Association for Computational Linguistics. Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, et al. 2020a. Open-domain conversational agents: Current progress, open problems, and future directions. arXiv preprint arXiv:2006.12442 . Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020b. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637 . Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International workshop on natural language processing for social media , pages 110. David K Sherman and Geoffrey L Cohen. 2006. The psychology of self-defense: Self-affirmation theory. Advances in experimental social psychology , 38:183242. Eric Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . ACL. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 . Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 29502968. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. DialoGPT: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536 . Wanzheng Zhu and Suma Bhat. 2021. Generate, prune, select: A pipeline for counterspeech generation against online hate speech. arXiv preprint arXiv:2106.01625 . 6472 A Task: Signaling Failure Figure 3: Screenshot from the Signaling Failure task. Each crowdworker is shown a 4-turn truncated piece of dialogue from the BAD dataset, that ends in an unsafe utterance, and instructed to label whether they consider the last utterance as NOT OK , and if so, write natural responses to signal to the other speaker that the previous message is NOT OK (see screenshot, Figure 3). Since we want responses that signal failure, we only keep responses if the crowdworker has marked the previous message as not ok. After collection, a separate task verifies whether the collected responses signals that its previous message was not ok with 3 other annotators. Using Krippendorff's alpha (Krippendorff, 2004) as inter-annotator agreement (IAA), the verification annotation task has a reliability coefficient of 0.213. This low value reflects both the overall skew of the dataset towards being "not ok" (about 70% of annotations overall), and the various ways in which workers interpreted what a good signaling response was (from calling out the type of offense, e.g. "this is sexist," to proposing a different opinion).
abstain abstain abstain abstain abstain objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective other abstain abstain abstain abstain abstain abstain method other other other abstain other other other other other objective other abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other