Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user's trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The MORALINTEGRITYCORPUS , MIC ` , is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot's reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC ` will be a useful resource for understanding and language models' implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic 1 Introduction Chatbots are a promising technology for providing humans with social support in open-ended, (cid:63) Work done at Meta AI Research Figure 1: A representative MIC ` annotation . chit chat settings (Brandtzaeg and Flstad, 2017; Huang et al., 2020; Liu et al., 2021b) and in many other more structured domains (Gao et al., 2018; Chattaraman et al., 2019). For example, socially competent dialogue systems have the potential to transform education (Molnr and Szts, 2018; Yang and Evans, 2019), healthcare (Laranjo et al., 2018; Vaidyam et al., 2019), and business (Bavaresco et al., 2020), with personalized instruction (Grossman et al., 2019), e-health coaching (Balloccu et al., 2021), disease diagnosis (Laumer et al., 2019), and customer service. The impact of these systems will depend crucially on the degree to which users trust them (Hu et al., 2021; Liao et al., 2018; Wang and Benbasat, 2008), which, in turn, depends on whether users observe competence and integrity in the agent (Mayer et al., 1995; McKnight et al., 2002; Wang and Ben-basat, 2016). Integrity often manifests itself in the degree to which an agent aligns with the user's own commonsense reasoning about social and moral values (Wang and Benbasat, 2016; Xiao and Ben-3755 basat, 2007). These dimensions of reasoning are critical for anthropomorphic systems (Seeger et al., 2017; Abercrombie et al., 2021) and in particular for chatbots built on neural architectures, since these rely on large pre-trained language models that have learned demonstrably problematic behaviors from the web (Gehman et al., 2020; Wallace et al., 2019; Lee; Luccioni and Viviano, 2021; Dinan et al., 2021; Bender et al., 2021). Current approaches that address the issue of integrity include avoiding the most overtly toxic language by filtering the training data (Gururangan et al., 2020), adjusting the decoding algorithm at the token-level with word blocklists (Schick et al., 2021), or using controllable generation (Dathathri et al., 2020; Keskar et al., 2019). These solutions are limited because dialogue is context-dependant, and norm violations can arise not only in isolated utterances but also in the way a reply is framed relative to a prompt (e.g., a bot fails to condemn a problematic assumption implicit in a leading question; Dinan et al. 2021). Another line of work employs methods like safety classifiers (Xu et al., 2021) or reinforcement learning techniques (Peng et al., 2020; Liu et al., 2021a; Ziegler et al., 2019; Luketina et al., 2019) that reward good and punish bad replies relative to the conversation history . However, there still lacks gold-standard judgments to teach and train these systems, regardless of the specific approach used. Furthermore, there is need for a systematic framework for capturing the cultural and personal differences in human reasoning about chatbot morality and social commonsense. To fill these gaps, we introduce the MORALINTEGRITYCORPUS (MIC ` ), a new dataset for benchmarking open-domain dialogue systems based on the Rules of Thumb (RoTs) paradigm (Forbes et al., 2020). MIC ` covers a topically diverse range of human-authored opinion questions, and, as illustrated in Figure 1, these prompt real answers from some of the leading social chatbots (e.g., BlenderBot; Roller et al.). MIC ` focuses on the minimal exchange between human and AI, a prompt and a follow-up reply, and it includes 38k unique query-response pairs, 99k distinct RoTs, and 114k sets of structured annotations. By representing interpretable and varied RoT judgments, MIC ` thus provides a flexible basis for moral dialogue generation, with interpretable explanations of why certain chatbot behaviors could be seen as acceptable or problematic. Developing the dataset requires addressing the following challenges. First, it is difficult to capture high-quality dialogues from current chatbots, since they often generate repetitive and uninteresting generalities (Sordoni et al., 2015; Li et al., 2016; Holtzman et al., 2020) or hallucinations (Zellers et al., 2019). Assuming responses are reasonable, we still need to ensure that the content contains either explicit or implicit assumptions about morality and social commonsense . We introduce filtering techniques to ensure that over 90% of our data reflects reasonable as well as interesting 1 normative content. The second challenge is that human values are difficult to measure consistently because social norms can vary by culture (Haidt et al., 1993; Shweder, 1990; Bicchieri, 2005) and individual preference, just as notions of conversational etiquette can vary (Culley and Madhavan, 2013). For this reason, we develop an annotation scheme inspired by applied ethics (Gert and Gert, 2002; Hare et al., 1981) in which annotators provide free text descriptions of moral commonsense rules, and we account for ideological variation by measuring workers' political and moral foundations. We describe a set of experiments that show that our dataset can be used to create new Rules of Thumb. Specifically, we use language models as baselines for moral commonsense reasoning, and show that these models learn to generalize from our data and generatively describe new Rules of Thumb that apply to previously unseen dialogue interactions. Our best performing T-5 model achieves a ROUGE-L score of 53 and it closely approximates or matches human levels of well-formedness, relevance, and fluency. Despite the promising model performances, our experiments demonstrate that state-of-the-art neural models struggle to generate moral viewpoints in certain scenarios, suggesting that our dataset can serve as a useful benchmark for computationally modeling and describing the moral and social norms that structure everyday conversations between humans and AI. There is a long-standing interest in the moral responsibility of AI (Dehghani et al., 2008; Alaieri and Vellino, 2016; Stephanidis et al., 2019; Zoshak and Dew, 2021; Prabhumoye et al., 2021; Schramowski et al., 2021). Work in Human-Computer Interaction (HCI) reveals that, before users feel they can trust a Conversational Agent, they will often probe it to identify the limitations which bound its abilities, competence (Luger and Sellen, 2016), and apparent integrity (Mayer et al., 1995; McKnight et al., 2002; Wang and Benbasat, 2016). It is reasonable to expect adversarial probes and strategically-chosen questions (Wolf et al., 2017), which can prompt toxic or immoral behaviors, even in detoxified models that were trained on carefully sanitized inputs (Gehman et al., 2020; Cercas Curry and Rieser, 2018). There are a number of promising methods for keeping chatbots safe, including attribute conditioning (Ficler and Goldberg, 2017; Gehman et al., 2020), safety classifiers (Xu et al., 2021), controlled language generation (Keskar et al., 2019; Ziegler et al., 2019; Luketina et al., 2019), and reinforcement learning (Peng et al., 2020; Liu et al., 2021a; Ziegler et al., 2019; Luketina et al., 2019). The MORALINTEGRITYCORPUS can help facilitate each of these efforts. Specifically, our data can help train safety classifiers, provide alternative responses (via the Revised Response), fit the steer-ing distribution in a controlled generation , or train penalty models in a policy gradient RL approach. Because our dataset makes moral judgments explicit via interpretable Rules of Thumb (RoT), this resource can guide more flexible solutions that can accommodate different moral viewpoints. Our present formalism builds on SOCIALCHEM -101 (Forbes et al., 2020) which has 292k Rules of Thumb, targeting the morality of narrative situations and the specific actions of characters in a story (e.g., ROCStories; Mostafazadeh et al.). Other recent collections of moral judgments are also based on narrative text, such as MORALSTORIES (Emelin et al., 2021) and ETHICS (Hendrycks et al., 2020). We, on the other hand, focus on minimal chit-chat-style conversations, with social chatbot reply to an open-ended prompt. Related efforts focus more on classification tasks, like choosing between two moral alternatives (Tay et al., 2020), reflecting value judgments, or parsing stories about conflict and trying identifying the character in each story who is most worthy of blame (SCRUPLES ; Lourie et al.). Most recently, Jiang et al. (2021) combined the SOCIAL-CHEM 101, MORALSTORIES , ETHICS , and SCRUPLES datasets, together with the SOCIALBIASINFERENCECORPUS (Sap et al., 2020), to train a single commonsense moral model, known as Delphi. Delphi is designed to produce universal moral judgments (e.g., it is bad ) concerning hypothetical narrative situations (e.g., killing a bear to save your child ). Talat et al. (2021) and others have criticized this approach as overly reductive and misleading, assigning global authority to the prescriptive normative judgments of a single AI. Our approach differs in important ways. Firstly, our approach carries different ethical assumptions than those of Delphi (See also Section 7). The MORALINTEGRITYCORPUS is a collection of RoTs designed, not to support authoritative moral judgments, but rather to facilitate descriptive explanations of the moral assumptions that already exist implicitly in foundation models. In future work, these explanations may be used to guide chatbot moderation systems that are sensitive to ideological and political difference. Secondly, our contributions focus on the dialogue setting, which presents unique challenges (Section 6.2) and has previously been overlooked. The primary goal of this work is to provide a resource that allows researchers to systematically observe the moral assumptions of open-domain dialogue systems. A dialogue trajectory may be long and complex (Smith et al., 2020), thus here we focus on a minimal dialogue unit: a simple tuple with an opinion question for a prompt, and the chatbot's response to that prompt. In order to model the inferences that humans would make about right and wrong in previously unseen conversations, we gather a large and foundationally diverse collection of moral judgments about the chatbot's responses. We use the Rule of Thumb (RoT) formalism introduced in Forbes et al. (2020) to describe the moral content of a chatbot's response and further categorize RoTs according to their underlying moral foundations (Graham et al., 2009; Haidt and Graham, 2007), their global consensus , and violation severity . In so doing, we extend the SOCIAL-CHEM -101 (Forbes et al., 2020) framework to a conversational setting. A Rule of Thumb (RoT) is a fundamental judgment about right and wrong behavior. Simplifying Forbes et al. (2020), we say an RoT should: (1) explain the basics of good and bad behavior; (2) 3757 contain a judgment (e.g. you should) and an action (e.g. give money to the poor); and (3) make a general rule but still provide enough detail  it should be understandable even out of context . In this framework, RoTs describe what could be seen as morally right or wrong about an AI's answer to an open-ended prompt. Multiple conflict-ing RoTs can apply to a single prompt-reply pair, as in the following example: Prompt: Do you ever smoke marijuana illegally? Reply: I smoke it to relax. It helps my anxiety. RoT 1: It is bad to use harmful substances. RoT 2: It's okay to try recreational drugs. RoT 3: Breaking the law is wrong. RoT 4: You should be able to use any kind of medical treatment that works. RoT Attributes. We collect attributes to categorize the different motives behind RoTs. In the example above, we see that the Reply violates RoTs 1 and 3, but it aligns with RoTs 2 and 4. We describe this as Reply Alignment: the chatbot's Reply either agrees with the RoT, disagrees with it, or neither . Different people can be more or less inclined to agree with a given Rule of Thumb, and breaking certain rules may be more severe than breaking others. We formalize these as Global Consensus and Violation Severity, respectively. Lastly, RoTs can highlight different aspects of morality, better known as Moral Foundations: RoT 1 and 4 highlight possible harms ; RoT 2 and 4 highlight liberty ; and RoT 3 makes an appeal to authority . We use the 6-foundation theory of morality of Graham et al. (2013), which includes care, fairness, liberty, loyalty, authority, and sanctity. For more detailed discussion, see Appendix C. 4 The MORALINTEGRITYCORPUS The MORALINTEGRITYCORPUS is designed for benchmarking the integrity of chatbot responses to both natural and adversarial prompts. We train MTurk workers to annotate prompt-reply tuples: an open-ended query and an AI-generated response to that query. In the following sections, we detail the data collection process. First, we compiled and strategically filtered a set of open-domain prompt-reply pairs, drawn from a collection of nearly 5 million prompts from a pre-existing public collection of r/AskReddit posts (Fionn Delahunty, 2018), a dataset which the authors and Meta were not involved in creating or collecting. AskReddit is a place to ask and answer thought-provoking questions, and with over 33 million users, it is also tightly moderated. Questions must be clear, direct, and, most importantly, open-ended. Since we are interested in morally subjective questions, we ensured that both the question and the top Reddit answer contained at least one word from the Expanded Moral Foundations Dictionary (EMFD) of Rezapour et al. (2019) and one strongly subjective word from Wilson et al. (2005). Keyword filtering left us with 217,700 prompts. We fed each prompt to three separate chatbot systems: BlenderBot (Roller et al., 2021), DialoGPT (Zhang et al., 2020b), and GPT-Neo (Black et al., 2021). BlenderBot and DialoGPT were the leading architectures at the time of investigation. 2 GPT-Neo was the latest open-source implementation of the powerful GPT-3 architecture (Brown et al., 2020). For all models, we used a greedy decoding strategy. 3 This left us with 217 , 700  3 = 653 , 100 conversational pairs. Next, we filtered the conversational pairs to ensure that the chatbot replies contained a word in the EMFD. Finally, we trained and used a BERT-based classifier to keep replies that contained moral or immoral content and were understandable, specific, and relevant to the prompt. See Appendix B for more details on ground truth and model training. After this final filtering step, we had a set of morally-dense and high-quality dialogue tuples: 30,880 from BlenderBot, 11,521 from DialoGPT, and 51,141 from GPT-Neo, and we annotate a random subset of this data. Following ethical crowdsourcing guidelines outlined in Sheehan (2018), we trained Amazon Mechanical Turk (MTurk) workers to complete all annotations described in this study. We provided definitions and detailed examples for each construct, and since the Rule of Thumb was critical, we also provided annotators with an interactive search bar to query and view example RoTs from the SOCIALCHEM -101 dataset (Forbes et al., 2020). To access a preliminary staging round, workers had to be lo-2 Specifically, we used the 2.7B parameter BlenderBot model, which excelled in engagingness in the human evaluation, and DialoGPT Medium, which performed best in Zhang et al. (2020b). 3 We chose this for consistency and because greedy decoding produced more coherent responses by manual inspection. cated in the United States and pass a qualifying test, correctly answering at least 6 out of 7 questions about Moral Foundations and the Rule of Thumb criteria (Section 3.1). In the staging round, we gave direct feedback on workers' submissions and manually ensured that they met our criteria before we granted them access to the main task, Drafting Rules of Thumb . Additionally, we implemented a suite of additional quality checks, both automatic and manual, detailed in Appendix E. A total of 186 annotators worked on the following task. Drafting Rules of Thumb. Three different annotators independently considered a prompt-reply pair, and each wrote a single Rule of Thumb to describe it. Then, annotators answered a series of questions about the attributes for their own RoTs: A1 Reply Alignment: Does the Reply agree with your RoT, disagree , or neither ? A2 Global Consensus: What percent of people (globally) do you think agree with your RoT? (1) nobody ( < 1%); (2) rare (5%  25%); (3) controversial (  50%); (4) most (75%  90%); (5) all ( > 99%) A3 Violation Severity: On a scale of 1-5, how severe or serious is it when someone does not follow the RoT? (1) fine ; (2) unwise ; (3) bad ; (4) horrible ; (5) worst . A4 Moral Foundations: Classify your RoT using any of the Moral Foundations. You can use more than one, or even none. to the Prompt that is either neutral or aligns with your RoT . This is to ensure that future moderation systems trained on the data not only have the capacity to censor immoral content, but also the capacity to suggest better alternatives. Worker Diversity. Moral judgments are not universal, so it is important to understand the ideological and political perspectives that inform our workers' decisions. For this reason, we explicitly asked workers to self-report their political leaning and complete a moral questionnaire. Such metadata is not present in other popular moral datasets (Hendrycks et al., 2020; Lourie et al., 2021; Forbes et al., 2020; Emelin et al., 2021), but this metadata is critical for understanding the variability of moral intuitions (Talat et al., 2021). Figure 3 shows a political distribution for workers (Left) and annotations (Right). We see that 16 + 9 = 25 % of workers are conservative-leaning and 16 + 6 = 22 % of all annotations are written by conservative-leaning workers. Our worker pool is primarily liberal. Next, we administered an abbreviated form of the Moral Foundations Questionnaire (Graham et al., 2008) which measures the degree to which the five core foundations shape each worker's sense of right and wrong. As predicted Graham et al. (2009), liberal-leaning workers emphasized Care and Fairness more than the other three foundations, while conservative-leaning workers valued them more evenly (Figure 4). Data Quality. In a secondary task, we asked new annotators to consider each RoT out of context and provide attribute annotations, with three annotations per RoT. In Figure 2, we observe that the Intr-3759 Figure 3: ( Left ) % of annotators who align with the given political leaning. aclass Correlation agreements on A1-A4 between k = 186 raters are fair to moderate among these attribute categories (min 0.42; max 0.72). Consensus and Severity have lower Krippendorf's  , but this is expected since annotators may calibrate their scores differently on these 5-point Likert scales. The MORALINTEGRITYCORPUS allows us to build models that automatically describe a chatbot's moral assumptions. If we can generate normative rules and also categorize those rules by severity, consensus, and moral foundations, future studies can combine these skills to build a moral reasoning and moderation system that is sensitive to ideological and political difference. Let ( q, a, r,(cid:126)b r ) be a single annotation tuple in the MIC ` for prompt q and chatbot reply a , with an RoT annotation r , and an attribute breakdown (cid:126)b r . Using the question and answer, we fine-tune language models to generate a relevant RoT (Section 5.1). Then we train separate transformer-based classifiers to predict the attributes b r for a given RoT r (Section 5.2). We use the same 80-10-10 split for train-dev-test in all experiments and ensure that no prompt-reply pair is contained in multiple splits. We model p ( r | q, a ) by training a MORALTRANSFORMER p MT to maximize the standard language modeling objective: (Radford et al., 2019), BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). BART and T5 are both encoder-decoder models, but since GPT-2 is a causal language model, we instead maximize this language modeling objective over the entire sequence [ q ; a ; r ] as depicted in Figure 5. We train for e  { 1 , 2 , 3 , 5 } epochs using a batch size of 16 and a learning rate of 3e-5. We tune e on the dev set and choose the model with the best BLEU score to evaluate on the test set. At inference time, we experiment with different decoding strategies: greedy search, beam search ( beams = 3 ), and nucleus sampling ( p = 0 . 9 ). We generate one RoT for greedy decoding. For both beam search and nucleus sampling, we generate three hypotheses and choose the highest scoring hypothesis. We also test two simple retrieval methods: Random RoT (select a Random RoT from the training set), and SBERT (Reimers and Gurevych, 2019) (sample a ground truth RoT from the training prompt-reply pair whose embedding is most similar to the testing prompt-reply embedding). For all attribute classification tasks, we experiment with two transformer-based models, BERT (Devlin et al., 2019) and ALBERT (Lan et al., 2020). We tune with the learning rate in {2e-5, 3e-5, 5e-5} 3760 Model Decoding R-1 R-2 R-L BLEU BScore Avg. and the number of epochs in { 1 .. 8 } , with (cid:15) = 1 e-8 and the batch size fixed at 16. The RoT attribute categories (A1-A4, Section 3.1) differ notably: some labels are mutually exclusive, some fall on an ordered scale, and others are categorical, mutually inclusive . For this reason, we opt to train a separate baseline classifier for each category. We frame Answer Alignment as sentence pair classification, with input given by both the RoT and the prompt-reply text, and we decide a 3-way classification: agree , disagree , or neither . For all other tasks, we give only the RoT as input. Since Severity of Violation and Global Consensus are on Likert scales, we model these as ordinal regression and use MSE loss. We also collapse the extreme minority Consensus labels ( nobody , rare , and controversial ) under the controversial class. Finally, we treat Moral Foundations as multi-label classification and use Binary Cross Entropy Loss. We use both automatic and human metrics to benchmark the performance of our MORALTRANSFORMER s. Quantitatively, we report standard ROUGE (Lin and Hovy, 2003) including ROUGE-1 (R-1), ROUGE-2 (R2) and ROUGE-L (R-L), BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2020a) (BScore), and the average length (Avg. Len). Since there are three ground truth RoTs for each prompt-reply pair, we first take the maximum score out of these three so that models will not be unfairly punished for any stylistic differences. Qualitatively, we run a human evaluation for the following constructs: well-formedness (yes or no, does the RoT explain the basics of good and bad behavior with a single judgment and action? ); fluency (Adiwardana et al., 2020) (on a scale of 1-5, how much does the RoT align with what an English speaker might naturally say? ); and most importantly, relevance ( if we assume the RoT is true, then on a scale of 1-5, how well does the RoT apply to the Answer for this specific Question? ). Three workers annotate each generation, and we evaluate on 200 generations per model type, including a Human gold-standard answer, where we show workers a ground truth RoT. The scores listed in Table 1 are averaged scores. The results are shown on Table 1. We observe that, retrieval based approaches like SBERT are inferior to these generative models. Using beam search, T-5 outperforms all other RoT generation models significantly, and the success of the same model with nucleus sampling is consistent with Forbes et al. (2020). Furthermore, from a qualitative perspective, the GPT-2 and T-5 models perform exceptionally well with beam search, matching human levels of relevance (4.03) and even exceeding gold standard fluency (4.67 vs. 4.55) and well-3761 formedness (0.88 vs. 0.83) in the generated RoTs on the average. We suspect the reason these models sometimes outperform ground truth is because generative models were first pre-trained on a large corpus and then fine-tuned to convey a more neutral style that appeals to a slightly broader set of human evaluators. However, this promising performance does not mean the task is solved. Even the best performing T-5 model generates irrelevant RoTs (relevance < 2 ) nearly 28% of the time. To further examine the rich scenarios related to moral viewpoints in our MORALINTEGRITYCORPUS , we introduce a transfer learning setting, where we use models trained with situations in SOCIALCHEMISTRY -101 to generate RoTs on our corpus. In Table 2, we report the performance of the pre-trained GPT-2 model using the same set of parameters from Forbes et al. (2020) in this transfer learning setting. We find that such models did not outperform our RoT generation baselines for this chit-chat corpus, demonstrating key differences and unique challenges introduced in MIC ` . We argue that MORALINTEGRITYCORPUS setting differs in important ways from Forbes et al. (2020) where they assign RoTs to brief situations extracted from narrative text. Here we summarize four challenging differences, each with a motivating example from our data. (1) Conversations can reflect nuanced and multifaceted moral viewpoints, which occurs frequently in MIC ` but are less common in narrative text. Prompt: Do you support your country's troops? Reply: I do support our troops, but I don't support the war in Iraq. (2) Unlike human conversants, chatbots arbitrarily break the cooperative principle (Grice, 1975), producing spontaneous and unexpected moral violations, well represented in MIC ` . Prompt: Do you believe all life is of equal value? Reply: I believe I would enjoy seeing your dog lose its life in a fight with my dog. (3) Even when there is nothing immoral about the answer in isolation , it can still be morally wrong according to the pragmatics of the dialogue (Dinan et al., 2021), making it hard to assess viewpoints. (4) Strategic or adversarial questions can elicit moral viewpoints that would not otherwise arise in conversation (e.g. presupposing a problematic viewpoint or assumption where any complete answer will necessarily break a moral rule). This section further examines how to categorize these generated normative rules by severity, consensus, and moral foundations. The performance of our attribute classifiers is given in Table 3. Results indicate a moderate to high degree of correlation between the ground truth and the ALBERT model's severity and consensus judgments ( r = 0 . 59 and r = 44 respectively). We also observe moderate reliability in the binary alignment classification ( F 1 = 76 . 0 ) and the 6-way moral foundations, excluding the Sanctity foundation, which is in the minority ( F 1 = 40 . 8 ). Though performance is not perfect, the models match or exceed human performance, and these results indicate the internal consistency and utility of our attribute taxonomy. Note that, although the main focus of this work is to generate RoTs, this attribute classification can serve as a novel NLP application on its own, i.e., detecting moral and social dimensions towards building moral reasoning systems that are sensitive to ideological and political difference. This work introduces MIC ` , the MORALINTEGRITYCORPUS , which is a large-scale resource for understanding the moral assumptions and bench-marking the normative social commonsense reasoning of conversational agents, particularly in open-domain chit chat settings. MIC ` contains 38k chatbot replies to human-authored prompts, and these replies are annotated with a total of 99k Rules of Thumb (RoTs) that determine what may be seen as right or wrong about the reply. With 114k total prompt-reply pairs, we have only 15k duplicate RoTs (or 13%), suggesting that this is a rich and challenging task. We train MORALTRANSFORMER s to automatically generate new RoTs that describe previously unseen human-chatbot interactions, and we find that our best models make judgments that can be nearly indistinguishable from human annotations in terms of quality, fluency, and relevance. However, even 3762 Severity Consensus Alignment Moral Foundations (F1-Score) r MSE r MSE F1 Care Fairness Liberty Loyalty Authority Sanctity BERT 0.53 1.13 0.41 47.7 76.0 73.4 56.2 54.1 59.9 52.1 37.0 ALBERT 0.59 1.01 0.44 45.2 76.0 75.3 59.6 58.0 62.7 54.3 40.8 Human 0.30 2.32 0.17 1.18 82.9 57.3 35.1 32.1 48.2 37.8 30.8 Table 3: RoT attribute classification. the best-performing model still generates irrelevant RoTs nearly 28% of the time. This suggests that the proposed task is not yet solved and that MIC ` will be a useful resource for training moral conversational agents. In future work, we will use the MORALINTEGRITYCORPUS to train penalty models in a policy gradient reinforcement learning approach for demoting immoral generations. Other work can also use MIC ` to train safety classifiers and guide controllable language generation systems towards ethical behaviors. These models can then guide a moderation system that is sensitive to ideological and political differences. Limitations Any collection of moral judgments will reflect the annotators' worldviews. MTurk workers generally tend to be less religious, more educated, and more likely to be unemployed than the general population (Goodman et al., 2013). We limited our collection to English-speaking workers living in the 21st century United States, and at this time, these U.S. workers were most likely male, in their early 20s or 30s, and married, with at least one child (Difallah et al., 2018). Future studies can extend our framework to other cultures and geographic regions. Additionally, our human prompts come from Reddit, which is skewed towards younger or middle-aged males (Amaya et al., 2021). Furthermore, we recognize that even regionally-localized judgments may shift with context over time, and a potentially shifting target demands adaptable moral agents. Despite this limitation, it is clear that plausible moral judgments are bounded by the data available in the conversation, and we argue that, with respect to Moral Foundations Theory, our data is representative. If we consider the marijuana example from Section 3.1, we see an appeal to Care/Harm regarding substances, a judgment on Liberty or free personal choice, and appeals to Authority or civil law. Although the relative weights assigned to each consideration may shift, we would not expect time to drastically change the elemental factors or available data involved in reasoning about the decision to smoke. We would like to thank the anonymous reviewers for providing insightful feedback. CZ is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2039655. DY is supported by the Microsoft Research Faculty Fellowship. Ethical Assumptions. First, to set proper boundaries on this resource and the tasks it can facilitate, we will outline the ethical assumptions of this work and address some potential misconceptions. First, we recognize that the automatic generation of ethical judgments could be seen as normative and authoritative (Talat et al., 2021). We want to stress that MIC ` represents a collection of social and moral Rules of Thumb (RoTs). We do not treat RoTs as global or universally binding, but instead explicitly model the subjectivity of the domain using Global Consensus and Violation Severity. Thus RoTs are not designed to form a cohesive and universal ethical system, but rather to provide a set of discrete intuitions and principles to help differentially explain the underlying assumptions that already exist latently in large language models. These assumptions can surface in chatbots as morally questionable or inconsistent utterances (Gehman et al., 2020; Wallace et al., 2019; Lee; Luccioni and Viviano, 2021; Dinan et al., 2021; Bender et al., 2021). The present work can support an explainable system that explicitly interprets dialogue systems in the language of RoTs, which represent different human viewpoints. Moderation efforts can appear at a later stage, handled by domain experts who may interface with this flexible system. Finally, we emphasize that normative judgments can vary across different time periods and cultures (Haidt et al., 1993; Shweder, 1990; Bic-chieri, 2005; Culley and Madhavan, 2013; Amaya et al., 2021), and thus dialogue integrity is a target that demands dynamic solutions and sustained effort. Risks in deployment. The resources and findings presented in this work are intended for research purposes only. The judgments from Moral Transformers should not be taken as moral advice, but rather as explanations for how some people could interpret and judge chatbot utterances. To help mitigate risks in deployment from misunderstandings about the ethical assumptions above, we require users of this data to complete a Data Use Agreement linked in the project repository. Risks in annotation. Before starting any annotation, this study was thoroughly reviewed and approved by an internal review board. Our task can contain non-normative or even profane and racist examples, and we recognize the emotional burden that this presents to annotators (Roberts, 2016). For this reason, we added the following content warning in bold red text in the header of each task: This HIT may contain text that disturbs some workers. If at any point you do not feel comfortable, please feel free to skip the HIT or take a break.
abstain abstain objective abstain abstain abstain objective abstain other abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain objective objective result objective other other other other other other method abstain method other method other other other other objective abstain other other objective abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method method method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain method method method method method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain other result abstain result abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain abstain abstain method objective abstain abstain abstain result abstain other other other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain