Sarik Ghazarian, 1 Nuan Wen, 1 Aram Galstyan, 1 Nanyun Peng 1, 2 1 University of Southern California / Information Sciences Institute 2 Computer Science Department of University of California, Los Angeles {sarik, nuanwen, galstyan}@isi.edu, violetpeng@cs.ucla.edu Abstract Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems as they facilitate hyper-parameter tuning and comparison between models. Although recently proposed trainable conversation-level metrics have shown encouraging results, the quality of the metrics is strongly dependent on the quality of training data. Prior works mainly resort to heuristic text-level manipulations (e.g. utterances shuffling) to bootstrap incoherent conversations (negative examples) from coherent dialogues (positive examples). Such approaches are insufficient to appropriately reflect the incoherence that occurs in interactions between advanced dialogue models and humans. To tackle this problem, we propose DEAM, a Dialogue coherence Evaluation metric that relies on Abstract Meaning Representation (AMR) to apply semantic-level Manipulations for incoherent (negative) data generation. AMRs naturally facilitate the injection of various types of incoherence sources, such as coreference inconsistency, irrelevancy, contradictions, and decrease engagement, at the semantic level, thus resulting in more natural incoherent samples. Our experiments show that DEAM 1 achieves higher correlations with human judgments compared to baseline methods on several dialog datasets by significant margins. We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM. Our results demonstrate the potential of AMR-based semantic manipulations for natural negative example generation. Despite the effectiveness of large pretrained language models (Radford et al., 2019; Lewis et al., 1 Our proposed manipulations, data, and trained models can be found at https://github.com/PlusLabNLP/ DEAM 2020) for dialogue response generation (Zhang et al., 2020; Adiwardana et al., 2020; Ghazarian et al., 2021a), it is still challenging for the models to imitate human-human conversations and maintain conversational-level coherence. To better evaluate such models, recent works propose trainable automatic evaluation metrics to benchmark and compare the performance of dialogue models (Wu et al., 2020; Zhang et al., 2021). Most trainable automatic evaluation metrics focus on turn-level interactions, where they learn to assess the quality of one user-system utterance pair (Tao et al., 2018; Huang et al., 2020; Ghazarian et al., 2020). However, these metrics cannot appropriately model the whole conversation flow (Yeh et al., 2021), and thus are insufficient for dialogue-level evaluation. In this work, we focus on the automatic evaluation of the coherence of dialogues , which is under-explored. Coherence is a conversation-level metric that measures how well the utterances in a conversation are unified leading to a consistent interaction (Byron and Stent, 1998; Mesgar et al., 2020). Previous works pursue different models such as graph-based (Vakulenko et al., 2018; Zhang 771 et al., 2021) or text-based (Mesgar et al., 2020) approaches to develop automatic trainable coherence evaluation metrics. Those models take a contrastive learning approach, where they build binary classifiers to differentiate positive, or coherent examples from negative, or incoherent dialogues. Those classifiers are usually trained on datasets constructed by using human-human conversations as positive examples and applying text-level heuristic manipulations to generate incoherent conversations. The text-level manipulations directly change the structures of the conversation such as shuffling the order of utterances, replacing some random utterances from external conversations (Vakulenko et al., 2018; Mesgar et al., 2020; Zhang et al., 2021), as shown in the third dialogue of Figure 1. We posit that such text-level manipulations are too simplistic to adequately represent more nuanced coherence errors presented in the current state-of-the-art dialogue systems. For example, the second conversation in Figure 1 shows a human-system interaction from the FED dataset (Mehri and Esknazi, 2020), where the incoherence is much more subtle than the ones created by text-level manipulations. In this paper, we investigate manipulation techniques to generate negative samples that represent coherence errors more likely to happen in the state-of-the-art dialogue systems. To this end, we propose DEAM, a model that uses Abstract Meaning Representation (AMR) to apply semantic-level manipulations to generate negative examples. AMRs are intended to capture the meaning of a sentence by abstracting away irrelevant syntactic features. Thus, injecting targeted and controlled perturbations into an AMR is easy and can introduce semantic incoherence into the corresponding sentences. DEAM starts with parsing conversations into semantic AMR representations and then injects incoherence types that are usually observed in current state-of-the-art models into the AMR graphs. It concludes this process by translating the manipulated AMRs back to conversations as negative examples using a controllable generation model. A fine-tuned RoBERTa model is then trained on the created dataset to distinguish coherent and incoherent conversations as the evaluation metric. We propose DEAM, an evaluation metric that leverages AMR graphs and injects incoherence sources at the semantic level to generate incoherent conversations for training. We propose four manipulation strategies to represent four common incoherence sources of the state-of-the-art dialogue models: contradiction, coreference inconsistency, irrelevancy and decrease engagements. We empirically show that the model trained on our proposed manipulations significantly outperforms strong baselines in terms of correlation with human judgments. Moreover, DEAM is capable of distinguishing positive and negative examples generated by baselines that use text-level manipulations, whereas the opposite is not true  classifiers trained on text-level manipulations cannot detect negative examples generated by DEAM. This demonstrates the effectiveness of the semantic-level AMR-based manipulations. Automatic evaluation of open-domain dialogue systems has a multifaceted nature with many fine-grained quality aspects (Mehri and Esknazi, 2020). Turn-level aspects show the quality of the sys-tem's utterance given a dialogue context from different perspectives including appropriateness, relevance, engagement, and etc (Lowe et al., 2017; Tao et al., 2018; Ghazarian et al., 2020). Whereas, conversation-level facets such as coherence, diversity, informativeness take into account the whole dialog flow (Vakulenko et al., 2018; Zhang et al., 2021; Mehri and Esknazi, 2020). Dialogue coherence evaluation is pertinent to discourse coherence since a dialogue is counted as a multi-party discourse. Similar to discourse coherence, many original coherence evaluation metrics derived from the Centering Model for monitoring the local focus of utterances and their entities distribution (Grosz and Sidner, 1986; Miltsakaki and Kukich, 2004; Lapata and Barzilay, 2005). A group of studies assess the coherence of dialogues with respect to entities and dialogue acts (Cervone and Riccardi, 2020; Mesgar et al., 2020). Another inspected approach for dialogue coherence evaluation is to represent dialogue in a structured graph format where contextually dependent neighbor utterances or concepts are connected nodes in the graph (Vakulenko et al., 2018; Mesgar et al., 2020; Huang et al., 2020). Graph convolutional networks are used to complete this task. of the momentous and indelible components in automatic coherence evaluation. Some previous works construct such datasets by collecting human judgments (Higashinaka et al., 2014; Cervone and Riccardi, 2020). While many recent works rely on a more timely and costly affordable approach by automatically generating negative samples. The utterances of the coherent conversations from human-human interactions are manipulated by shuffling their order, inserting or replacing irrelevant utterances (Vakulenko et al., 2018; Mesgar et al., 2020; Huang et al., 2020; Zhang et al., 2021). In this work, we show that such changes can not truly represent machine-generated incoherent conversations. One work that is closely related to us and proposed abstract-level manipulations is Ghazarian et al. (2021b). However, their application domain is open-domain story evaluation, and they use story plot, rather than AMR, for the manipulation, which is more domain-specific. Our goal is to build an evaluation metric that measures the conversation-level coherence of dialogues. We follow the trainable evaluation metrics (Vaku-lenko et al., 2018) to formulate the evaluation as a classification task. We train the evaluator on positive (coherent) and negative (incoherent) conversations, and take the predicted probability for the positive class as the coherence score. As is discussed above, the main challenge for building a reliable metric is to obtain negative samples that can adequately represent the incoherence issues presented in advanced dialogue systems. To this end, we propose to generate negative examples by leveraging AMR-based manipulations. We then build a RoBERTa-based classifier as the evaluation metric by fine-tuning RoBERTa on the automatically generated training data. Figure 2 illustrates an overview of our proposed evaluation method. The first step of DEAM is to apply Text-to-AMR models to the conversations. Text-to-AMR or AMR parsing (Jin and Gildea, 2019; Xu et al., 2020; Zhou et al., 2021; Lam et al., 2021) that translates conversation texts to directed and acyclic AMR graphs containing relation edges between concept nodes (Banarescu et al., 2013) has been effectively accomplished by transformer-based models in a sequence-to-sequence (seq2seq) training fashion (Xu et al., 2020; Zhou et al., 2021). We use  Text-to-AMR AMR-to-Text RoBERTa  We could take the bus there. We then manipulate the AMR graphs (sec-tion 4.2) and back translate them into conversation texts to be used as negative examples for training the text-based coherence evaluator. Similar to AMR parsing, we use finetuned T5 3 that is shown to be effective for the AMR-to-Text generation task (Mager et al., 2020; Ribeiro et al., 2020). The challenge that automatic trainable evaluation metrics face is in providing training data that can appropriately replicate moderate to low quality conversations with incoherence sources that usually happen in the current dialogue models. The common solution is to apply manipulations to positive conversations. In this section, we summarize the baselines manipulations and state our proposed AMR-based perturbations. Baseline manipulations can be classified as: 1) Shuffling-based manipulations: In such manipulations, turns order (Vakulenko et al., 2018), sequence of speakers utterances (Mesgar et al., 2020; Vakulenko et al., 2018; Zhang et al., 2021), or the position of the first and second sections of conversations (Vakulenko et al., 2018) are swapped. 2) Insertion-based manipulations: This group of manipulations add incoherence sources by replacing (Mesgar et al., 2020; Zhang et al., 2021) or inserting (Mesgar et al., 2020) a random utterance from a randomly selected conversation. Each baseline metric fuses multiple manipulations, hence we use their citations (Vakulenko et al., 2018), (Mesgar et al., 2020) to easily refer them in later sections. AMR is originally proposed by Banarescu et al. (2013) as a semantic representation language that helps to abstract away the text from surface syntactic. Many abstract-level semantic information such as named entities, negations, questions, coreferences and modalities in the texts can be encoded by AMR graphs. These potential capabilities of AMR make it lucrative in many semantic-related NLP tasks such as summarization (Liao et al., 2018) and machine translation (Song et al., 2019). Conversations between two interlocutors contain many semantic details that can be captured by these graphs. Therefore, we explore AMR features' usage in the dialogue systems evaluation task by manipulating the AMR graphs of coherent conversations, each manipulation reflecting a specific reason for incoherence in dialogue systems. Figure 3 demonstrates a linearized version of an utterance AMR graph. In AMR graphs, entities and concepts are shown as nodes and their relations are depicted with various relation edges (Banarescu et al., 2013). Each AMR concept is either a word, or a PropBank framesets keyword (Kingsbury and Palmer, 2002). The PropBank framesets with their predefined arguments are used to abstract away concepts from syntactic structures. As an example, located concept of PropBank framesets in Figure 3 comes with two arguments the subject ( i ) and the place ( school ). In DEAM, we pursue the idea of manipulating abstract-level semantic information extracted from AMRs to generate incoherent conversations. In this work, we mainly focus on four major logical flaws that could happen in state-of-the-art dialogue models such as cases when a chatbot contradicts its previously stated utterances, uses incorrect coreferences, responds users with irrelevant utterances, does not engage enough in the conversation. We explain each of these logical flaws in detail. One of the common issues that dialogue systems struggle with is directly or indirectly contradicting previous utterances in dialogue. To replicate this type of error, a contradicted version of a subgraph from the original AMR is copied to other locations. This negative form AMRs can be accomplished by directly adding polarity to the concepts or replacing concepts with their antonyms that hold Antonym , NotDesires , NotCapableOf , and NotHasProperty relations in ConceptNet (Speer and Havasi, 2012). After adding contradictions, the AMR-to-Text model will use the encoded context to output incoherent yet natural conversations. In the bottom right example of Figure 4, speaker B contradicts its previously stated opinion that badly effects the linkage of the utterances. The coherence of a conversation is preserved by the correct references of previously mentioned entities and words in the dialogue context. Pronouns in the conversation play an essential role in this regard. Coreferences in AMRs are presented as arguments ( ARG ) and all three different types of pronouns such as subjective, objective and possessive pronouns are shown in their subjective format. To disrupt the coreferences relations, we randomly replace some pronouns in the conversation's AMR with another pronoun or noun identified as ARG or operand ( op ) from the same conversation. After replacements, the AMR-to-Text model adapts other sections of the utterance accordingly and reassures us that outputs have natural look and correct grammar. The third utterance in Figure 4 demonstrates an example of coherence inconsistency which makes the utterance to be not logical. Random utterance substitution from other conversations is a simple way to inject incoherence sources in dialogues, which has been frequently used in prior work (Tao et al., 2018; Ghazarian et al., 2019; Mesgar et al., 2020; Zhang et al., 2021). Conversations with completely off-topic utterances are rarely generated by advanced dialogue models due to their ability in encoding dialogue history for continuing the conversation. We propose to apply irrelevancy sources to AMR graphs. We select some AMR items such as concepts, op s, ARG s and replace them with random items from other utterances. In this approach, the replacement items are not from randomly selected conversations but still, they do not fit well in their newly selected locations which hurts the coherence of the conversation. In Figure 4 watch is replaced with listen . The benefits of using AMR-to-Text model emerge here where some new adaptations (such as to ) have been augmented with new verb replacement to give the utterance a fluent look. In coherent conversations, speakers exchange opinions about different topics by stating detailed information, asking and answering questions. This coherence will be faded if one of the interlocutors evades to answer questions or talk in detail. In contrast to previous works that ignored this important feature, we augment such kind of incoherence sources into the negative sampling generation. In order to decrease the engagement of coherent conversations, we take the advantage of AMRs which are able to demonstrate detailed utterances and those containing questions. In AMR graphs, detailed utterances include more number of nested layers and concepts, ARG s and op s. Question-type utterances can be easily distinguished via amr-unknown concept notation and therefore with relying on AMRs the goal of decreasing engagement in the conversation is easily achievable. conversation: we select a multi-sentence utterance including amr-unknown concept and remove it and all its children nodes from the graph; 2) Remove the most detailed utterance in the conversation: the utterance having the largest depth in the graph is selected as the utterance with the most transferred information and all its children alongside its parent concept are removed from the graph; 3) Remove fine-grained information in the utterances: the main concepts' detailed information that are presented as ARG or op in the AMRs are randomly selected and eliminated from the graph. The higher-level concepts in the graph are preserved while its lower-level child nodes are deleted which makes the utterance not transfer its meaning and diminishes the linkage of topics. The question part in the third utterance of Figure 4 has been removed causing the coming utterances to not be completely sensible. We compare DEAM and its negative example generation techniques with baseline models and manipulations. We aim to have a data-driven analysis under three setups: Setup 1) : In this setup, we compare DEAM with baseline models by varying both the data manipulation strategies and the classification models. We fix the positive examples to be the same set of human-human conversations. Setup 2) : Since baseline models are trained on different datasets, we conduct pairwise comparisons between DEAM and each baseline evaluator by training on the baseline's dataset. Note that we only take the positive examples from the baseline's dataset, and apply different manipulations to get 775 Dataset size conv. negative examples to compose a balanced set for training. We also train different classifiers (DEAM vs. baselines) for the evaluators. Setup 3) : This setup is designed to show the effectiveness of different manipulations to generate negative examples. We fix the positive examples and the classifier (i.e., RoBERTa). We conduct our experiments on two crowd-sourced datasets, TopicalChat (Gopalakrishnan et al., 2019) and PersonaChat (Zhang et al., 2018). Both datasets are composed of conversations between Amazon Mechanical Turk (AMT) participants. In TopicalChat, AMT workers were supposed to have coherent and engaging conversations regarding the provided reading sets about different topics, while in the PersonaChat dataset coherent conversations were conditioned on the provided 1155 personas each including 5 personality description sentences collected via AMT. We take these conversations as coherent conversations. We follow DEAM's steps to generate and add a balanced number of incoherent conversations. Table 1 shows the train/valid statistics of the newly constructed datasets called TOPICAL _DEAM and PERSONA _DEAM. In the literature of automatic evaluation metrics, the prevalent way of assessing the evaluators' performance is to compare the correlation of their predicted scores with human judgments. FED (Mehri and Esknazi, 2020) and Interactive Evaluation of Dialog track of the Dialog State Tracking Challenge 9 (DSTC9) (Gunasekara et al., 2020) are two publically available benchmark datasets including human ratings on the coherence aspect of human-human or human-systems conversations. human-Mitsuku chatbot and 40 human-Meena (Adiwar-dana et al., 2020) chatbot. Humans have assessed the conversations from 11 conversation-level evaluation aspects including the coherence and overall scores. Each conversation in FED is judged by 5 distinct annotators. Coherence and overall scores are in the range of 0-2 and 0-4, respectively. In DSTC9 dataset 4 , AMT workers have rated 2200 conversations between invited participants and knowledge-grounded response generation models using the same 11 fine-grained conversation-level evaluation aspects. Coherence and overall scores that we use in our experiments are in the range of 1-3 and 1-5, respectively. In our experiments, we take the average of judgments for conversations with more than one annotator's ratings and compute the Spearman correlations between human evaluations and evaluator's generated scores. In our work, we train and run all the models on a machine with a GeForce RTX 2080 Ti GPU. We fine-tune RoBERTa-large pretrained model on TOPICAL _DEAM and PERSONA _DEAM datasets for three epochs and optimize parameters using Adam optimizer with 1e-5 learning rate. To conduct experiments in setup 1, we train Vakulenko et al. (2018)'s graph-based model for 128 epochs with 1e-5 learning rate. Mesgar et al. (2020)'s LSTM-based model is trained for 8 epochs with 5e-5 learning rate. We retrain DynaEval (Zhang et al., 2021) for 20 epochs. All baselines are trained using Adam optimizer. Due to not publically published models proposed by Vakulenko et al. (2018) and Mesgar et al. (2020), we need to retrain these models on their original datasets; Ubuntu (Lowe et al., 2015) and DailyDialog (Li et al., 2017); using the same hyperparameters published in the aforementioned papers to complete experiments in setup 2. We use DynaEval's published checkpoints to run experiments in this setup. In experimental setup 3, we start from TOPICALCHAT and PERSONACHAT datasets, and augment negative samples pursuing different manipulation techniques. We fix the evaluator and finetune RoBERTa-large model for 3 epochs with a 1e-5 learning rate. Since Vakulenko et al. (2018)'s proposed manipulations are in the entity level, we adapt the perturbations to the text level by replac-4 https://github.com/exe1023/ DialEvalMetrics 776 Model FED DSTC9 Coh Ovrl. ing the sequence of entities with a sequence of utterances substitutions to be acceptable by the RoBERTa model. Through our experiments, we report the Spearman correlation of evaluation metrics with human annotations under different experimental setups. Table 2 depicts the quantitative results for different evaluation models on both FED and DSTC9 datasets of experimental setup 1. According to the reported correlations, the superiority of DEAM shown in the last row versus other baselines is obviously recognizable. This superiority could originate from the subtle negative sampling technique. In experimental setup 1, manipulation techniques and models vary between evaluators, therefore we complete our investigation via experimental setup 2 by conducting one by one comparison of DEAM with each baseline model training each pair on the same dataset that the baseline model has been trained on. Table 4 shows the output of this type of pairwise comparisons separated each pair into one section. Even though most of the baseline models correlations increased, yet DEAM takes the lead. It is noteworthy that the correlation of DYNAEVAL reported in the original paper for FED dataset has decreased, this could be due to the less number of negative samples that we consider for each positive conversation and its major impact on this model's performance. Table 3 illustrates the results of experimental setup 3, where we fix both RoBERTa evaluator and TOPICALCHAT and PERSONACHAT original datasets and apply different manipulation techniques to add negative samples. Even though the correlation for baseline manipulations increased drastically, which shows the effectiveness of strong pretrained language models in better encoding conversations information used for the evaluation task, DEAM's performance is still higher. This interprets the beneficial effect of AMR-based manipulations. The positive slops of the regression line in Figures 5 and 6 between DEAM predicted coherence scores and human coherence and overall evaluations for FED dataset show the proposed manipulations superiority from a different angle. The distribution of baseline models predicted low scores for high-quality conversations and vice versa present their ineffectiveness in correctly distinguishing between low-quality and high-quality conversations. Next, we inspect the role of each of the four proposed manipulations in the metric's performance. We conduct an ablation study on TOPICALCHAT and PERSONACHAT datasets to assess the effectiveness of each manipulation. For each specific manipulation, we remove it from the list of possible 777 Model Manipulations Dataset FED DSTC9 Coherence Overall Coherence Overall Vakulenko et al. (2018) Vakulenko et al. (2018) UBUNTU 0.17 0.15 -0.04 -0.1 DEAM DEAM UBUNTU 0.18 0.25 0.16 0.15 Mesgar et al. (2020) Mesgar et al. (2020) DAILYDIALOG -0.36 -0.47 0.13 0.14 DEAM DEAM DAILYDIALOG 0.34 0.36 0.17 0.18 DYNAEVALDYNAEVAL EMPATHETIC 0.17 0.10 -0.01 -0.02 DEAM DEAMEMPATHETIC 0.48 0.47 0.20 0.20 Table 4: Pairwise comparisons between DEAM with the proposed AMR-based manipulations and different baseline models using their original datasets, manipulations, and models (setup 2). manipulations and try to randomly sample one up to three different manipulations to create negative samples. In Table 5, we witness an overall drop by eliminating each of the manipulations that indicates the positive impact of all of the manipulations on generating higher quality negative samples that are closer to the samples generated by state-of-the-art models and consequently the evaluator's accuracy. Removing irrelevancy and decrease engagement manipulations have the most detrimental impact on the metric, which suggests that many state-of-the-art models struggle with such issues. By eliminating these manipulations the model does not have access to such negative examples during training, which significantly limits its ability to detect such incoherences during inference time. On the other hand, omitting coreference inconsistency from the manipulations has the lowest impact on DEAM, specifically for DSTC9 dataset which can be interpreted as the state-of-the-art models are safer regarding such issues. We also note that the performance difference between DSTC9 and FED could be due to the long conversations in DSTC9 that mostly include very limited coreferences. We analyze the quality of DEAM versus baseline evaluators in terms of examining each model's performance to distinguish between positive and negative examples constructed leveraging various manipulations. Some examples are shown in Table 6 of Appendix. Figure 7 illustrates a heat map of the accuracy scores. X-axis and Y-axis show the manipulations used for creating training and testing datasets, respectively. As is expected the highest accuracies can be found on the diagonal where models have been trained and tested on datasets generated from pursuing the same manipulation techniques. The light-colored cells are mainly related to models trained on baseline manipulated data and tested on AMR-based perturbed data. This indicates that the baseline models trained on such 778 Figure 8: Statistics of different types of logical flaws observed in 50 randomly selected low-quality interactions between human and state-of-the-art dialogue systems (25 dialogues from FED and 25 from DSTC9) types of text-level heuristic manipulations can not perform well and indeed have a random guess on more challenging incoherent examples that are generated by DEAM. While the higher accuracies of DEAM model on baseline test datasets show its capability to more effectively distinguish between positive and heuristically created their counterpart manipulations' negative conversations. Our proposed manipulations in DEAM are directly influenced by the quality of Text-to-AMR and AMR-to-Text generation models. Even though finetuned T5 (Raffel et al., 2020) models used here have been shown to be effective (Ribeiro et al., 2020), there are still not per-fect and suffer from some errors. We conducted a quick analysis of different deficiencies in conversations obtained by AMR-based back-translations. Most of these flaws are due to the fact that in Text-to-AMR generations some syntactic information such as verb tense, passive type of sentences, are removed from the text due to the semantic-based structure of the AMRs. Table 8 in the Appendix shows such flaws. Ongoing work on improving AMR parsers and generators will lead to more robust AMR models, thus improving the quality of the proposed manipulations as well. In the end, we conduct an analysis to explore the coverage rate of our proposed manipulations in the test datasets. To accomplish this, we analyze different commonly occurring logical flaws by advanced dialogue models via randomly selecting 25 low-quality interactions from FED (Mehri and Es-knazi, 2020) and 25 poor dialogues from DSTC9 datasets. The low-quality scores specified by human annotators indicate various types of flaws in the conversations. Our analysis suggests that we can classify those flaws into distinct categories as demonstrated in Figure 8. Most of the frequently happening flaws have been covered in our work except not_sensibility showing the sensibility of the generated responses. We leave not_sensibility evaluation for future works as replicating such issues besides the AMR-based manipulations mostly requires external knowledge bases which is not the focus of this work. Reliable automatic trainable coherence evaluation metrics that can efficiently measure the dynamics of interactions between interlocutors are principally influenced by the quality of the training instances. We show that leveraging text-level manipulations can not adequately mirror the incoherence errors that current dialogue systems face. According to our study, DEAM can more effectively accomplish this task by relying on capabilities that AMR-based semantic perturbations and pretrained language models present. We leave the investigations regarding the effectiveness of AMRs for evaluating sensibility of the generated responses with taking into account knowledge bases for the future. We acknowledge the importance of ACM Code of Ethics and totally agree with it. We ensure that our study is compatible with the provided code, specifically in the terms of providing non-offensive dataset construction. In our proposed approach, we start from publically available human-human conversational datasets and attempt to apply our proposed manipulations at the AMR level. The main concern that arises here is the probability of generating offensive conversations from manipulated AMRs. The chance of such generations is faded with leveraging TopicalChat (Gopalakrishnan et al., 2019) and PersonaChat (Zhang et al., 2018) datasets that have been originally collected asking users to converse without profanity and inappropriate utterances. We should note that the possibility of perturbations that would have the possibility of generating objectionable outputs is not zero therefore we acknowledge there may be biases or abusive content via attacks that can be resolved by security trended studies which is out of this work's scope.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain objective abstain abstain abstain abstain abstain objective abstain objective objective abstain abstain abstain abstain abstain objective objective objective objective objective objective other other other other other other other other other other other other abstain abstain other abstain method method abstain abstain method abstain abstain other method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method method method abstain method abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain method abstain method method method method other abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain result abstain method abstain method abstain abstain abstain method