Role-oriented dialogue summarization is to generate summaries for different roles in the dialogue, e.g. , merchants and consumers. Existing methods handle this task by summarizing each role's content separately and thus are prone to ignore the information from other roles. However, we believe that other roles' content could benefit the quality of summaries, such as the omitted information mentioned by other roles. Therefore, we propose a novel role interaction enhanced method for role-oriented dialogue summarization. It adopts cross attention and decoder self-attention interactions to interactively acquire other roles' critical information. The cross attention interaction aims to select other roles' critical dialogue utterances, while the decoder self-attention interaction aims to obtain key information from other roles' summaries. Experimental results have shown that our proposed method significantly outperforms strong baselines on two public role-oriented dialogue summarization datasets. Extensive analyses have demonstrated that other roles' content could help generate summaries with more complete semantics and correct topic structures. 1 1 Introduction Dialogue summarization aims at compressing the main content of a long conversation into a short text. With the development of online conversation tools, the amount and length of conversation are growing up rapidly. Since a dialogue often contains complicated structure and ellipsis, it is time-consuming to read the whole dialogue. Dialogue summarization thus becomes valuable since it could extract the key point of a conversation and greatly reduce the time cost. This technique is widely used in customer service (Liu et al., 2019), meeting (McCowan et al., 2005), online chatting (Gliwa et al., 2019), etc. In a dialogue, each role has its own opinion and goal, and different roles exchange information or reach a consensus through interactions. Therefore, in addition to summarizing the whole dialogue, we could summarize the main content for each role. Lin et al. (2021) first define the role-oriented dialogue summarization task and provide a related dataset, CSDS. They define role-oriented dialogue summarization as grasping the main viewpoint of a given role from dialogue and mention the usage of role-oriented summaries in the customer service domain, e.g. , reflecting the user's requirements and evaluating agent service quality. Besides, role-oriented summarization is beneficial to other dialogue domains such as medical inquiry (Song et al., 2020) and court debate (Duan et al., 2019). labeling process (Song et al., 2020). They ignore the strong relativeness among summaries for different roles and thus fail to utilize the information from other roles to enhance the summaries. However, information from other roles is also crucial for this task. We summarize two cases where other roles' information helps: (1) Other roles' dialogue utterances could help enhance the informativeness of summaries. In Figure 1, utterance 7 ( Yes, it is OK normally. ) is the key utterance of the agent's content, expressing a confirmation to the user's question. While only extracting it makes the agent summary ambiguous since it lacks the confirming object ( JD can pay via wechat in blue). In this case, the agent summary needs to integrate the content from the user's utterance (utterance 6 in yellow) to enhance its informativeness. (2) Other roles' summaries could help judge the key content in the dialogue. In a dialogue, different roles often discuss the same topic. Therefore, considering the key content of the other role could help grasp the key content of a given role. As shown in Figure 1, the user summary contains a question about the payment (in red), and the agent summary contains the response to the payment question (in blue). If the summary of one role struggles in judging whether the discussion about payment should be contained in the summary, by referring to the summary of the other role, the summarization model could be more confident to include this information in the summary. Although we notice the importance of other roles' information, it is difficult to extract the key information from other roles through a simple multi-task framework. The main issue is that it could not judge which information from other roles is useful without modeling the interaction between different roles. Thus, in this work, we propose two interaction methods to obtain key information from other roles for enhancing role-oriented summarization. First, we apply a cross attention interaction to let each role decoder select the most useful dialogue utterances from other roles. Specifically, we calculate the Cross Attention for different roles' utterances separately and add a new Attention Divergence Loss to interactively share the cross attention distributions between different roles. Second, we apply a decoder self-attention interaction to let each role decoder obtain other roles' summary information. We develop an interactive mechanism between decoders to consider other role summary information embedded in the decoder states. A new Role Attention module is added to each role decoder, where the attention object is the hidden states of other role decoders. At last, we use the role attention result and multiple context attention results to predict the word probability distribution of the summary. Through these two modules, the model could acquire more precise information from other roles and provide better role-oriented summaries. To examine the effectiveness of our method, we conduct experiments on two dialogue summarization datasets (Lin et al., 2021; Song et al., 2020) with role-oriented summaries in different domains (customer service, medical inquiry). We apply our method on two widely-used summarization frameworks (RNN-based and Transformer-based). The results have shown that, compared with baseline systems and naive multi-task approaches, applying role interactions could significantly improve the quality of role-oriented summaries. Further analyses verify that our proposed method can help the model correctly attend to other roles' key information and generate summaries with more complete semantic and correct topic structures. The main contributions of this paper include: (1) We are the first to enhance role-oriented dialogue summarization by focusing on other roles' key information. (2) We innovatively design two role interaction methods to obtain other roles' key information useful for generating summaries. (3) Experimental results on two datasets have shown that our method could lead to considerable improvements. Besides, our method has good generalizability since it works on multiple baseline frameworks. Dialogue summarization has been studied in various domains, e.g. , meeting (McCowan et al., 2005; Janin et al., 2003), daily chatting (Gliwa et al., 2019; Chen et al., 2021), customer service (Liu et al., 2019; Zou et al., 2021), and medical inquiry (Song et al., 2020; Krishna et al., 2021). Considering the particularity of dialogue, many studies try to improve the dialogue summarization performance by focusing on dialogue-specific features (Feng et al., 2021), such as topic information (Chen and Yang, 2020), discourse structure (Chen and Yang, 2021), coreference information (Liu et al., 2021) 2546 and speaker information (Lei et al., 2021; Zhu et al., 2020). However, all the above studies focus on summarizing the whole dialogue. Only a few studies pay attention to role-oriented summarization, which aims to summarize the main content of a single role in the dialogue. A relative task is focused meeting summarization (Wang and Cardie, 2013; Mehdad et al., 2014; Zhong et al., 2021). It aims to summarize a specific part of the meeting dialogue, while role-oriented summarization focuses on a single role, and the relationship between different roles is much closer. Tamura et al. (2011) focus on contact center dialogue summarization, but they only extract salient sentences from the dialogue and do not summarize for different roles. Due to the lack of labeled data, Zhang et al. (2021) propose an unsupervised method to generate summaries for the customer and the agent under a variational auto-encoder framework. As for supervised methods, there are only two datasets available for training. Lin et al. (2021) propose a customer service domain dataset named CSDS, where each dialogue has an overall summary and two role-oriented summaries for user and agent. They train two separate models for generating user summaries and agent summaries. Song et al. (2020) provide a medical inquiry dialogue summarization dataset where each dialogue has two extractive summaries for the patient and the doctor. They train a sequence labeling model to extract summaries for these two roles. Compared with these approaches, to the best of our knowledge, we are the first to enhance role-oriented summarization by explicitly considering other roles' critical information. Interactive decoding is a mechanism to share information between different decoders in the decoding process. Zhou et al. (2019) propose this mechanism and use it on machine translation to simultaneously decode from both left-to-right and right-to-left. Wang et al. (2019) and Liu et al. (2020) further utilize it on more complex machine translation tasks, including multilingual translation and speech translation. In this work, we first use the interactive decoding mechanism on the summarization task to decode summaries for different roles, aiming at utilizing other roles' summary information for summarization. Besides, we also propose an interaction method on cross attention to utilize other roles' critical dialogue utterance information. Given a dialogue D containing m utterances { u 1 , ..., u m } and p speakers S = { s 1 , ..., s p } , the role-oriented summarization task aims to generate a summary y k for each speaker s k . Each utterance u k consists of a speaker role r k  S and related content. By concatenating all the utterances and related speaker roles, we achieve the final input { x 1 , ..., x n } . Note that since both datasets used in this work have two speakers, one asking questions and one giving answers, we thus use y user and y agent to represent two role-oriented summaries in the following illustration 2 . In a traditional encoder-decoder framework for dialogue summarization, the encoder hidden states represent the semantic information of input dialogue utterances, and the decoder hidden states contain the information used to generate summaries. To fully exploit the information from other roles, we apply two role interactions on the attention module of both hidden states. We present the structure of our method in Figure 2 and introduce the details of interactions in the following paragraphs. Our method is constructed based on a multi-task framework where an encoder is used to encode dialogue utterances and two role decoders (user decoder and agent decoder) are used to decode user summary and agent summary. First, the input { x 1 , ..., x n } is sent to an encoder (omitted in the figure for simplicity) and the encoder outputs the context hidden representation { h 1 , ..., h n } . In the decoding phase, to calculate the cross attention results for different roles separately, we use User Mask and Agent Mask to split the context information into user context H enc u and agent context H enc a . H enc u contains the hidden representation of all user utterances, and H enc a contains the hidden representation of all agent utterances. The cross attention module extracts the most useful information from the context according to the temporary decoder state. Here we modify the module to attend to different role contexts separately. 2 Here we need to point out that our method could also apply to dialogues with more than two speakers. Taking user decoder as example, at step k , we use the hidden state of user decoder h user k to attend to user context H enc u and agent context H enc a , obtaining two attention distributions att uu,k , att ua,k and context attention results c uu,k , c ua,k . Both context results involve generating summaries. The process is the same with agent decoder, where two attention distributions are noted as att au,k , att aa,k . Since existing models are poor at extracting important information from other roles, it reflects in incorrect cross-role attentions att au,k (agent decoder to user context) and att ua,k (user decoder to agent context). Meanwhile, the same-role attentions att uu,k (user decoder to user context) and att aa,k (agent decoder to agent context) are learned better since most information of role-oriented summaries comes from the given role's utterances. Thus we want to use the same-role attention to guide the cross-role attention. As different roles often discuss the same topic in one dialogue, the accumulated cross attention distribution for user decoder and agent decoder on the same role's utterances should be similar. A new Attention Divergence Loss is added to constrain this attention similarity as: L att  user = KL(Avg( att au ) || Avg( att uu )) L att  agent = KL(Avg( att ua ) || Avg( att aa )) By minimizing these two losses, the agent decoder attends to user utterances as the user decoder does, and the user decoder attends to agent utterances as the agent decoder does. Two role decoders interactively learn to focus on the key information of the other role in dialogue utterances. Since the decoder calculates the hidden states that could help predict summaries, the hidden states must contain much important information of summaries. We thus try to exploit the information embedded in other role decoders. Specifically, for user decoder, at time step t , we achieve the decoder hidden states h user t and use a Role Attention module to weigh the last t hidden states of agent decoder { h agent1 , ..., h agent t } 3 . The role context information r user t is obtained by adding all the agent hidden states with their weights, and it helps generate the probability of next word  y user t for user summary. The calculation formulas are given as: r user t = Attn( h user t , h agent1: t ) p ( y user t ) = F ( h user t , r user t , c uu,k , c ua,k ) The function F includes an MLP layer to fuse different information and a softmax layer to predict the vocabulary probability distribution. The process is the same with the agent decoder, and two decoders decode interactively. In the training phase, we use the teacher-forcing method to jointly train two role decoders and use the Negative Log-Likelihood loss to optimize. The 3 Since two decoders decode simultaneously, at step t , the other decoder could only provide the states from step 1 to t . L nll =  (   | y user | (cid:88) i =1 log P ( y user i | y user <i , y agent <i , D )+ (1   )  | y agent | (cid:88) i =1 log P ( y agent i | y agent <i , y user <i , D )) is a hyper-parameter for balancing the weights of different summarization tasks. Besides, we add the attention divergence loss to constrain the attention distribution, and the total loss is calculated as: L = L nll +  ( L att  user + L att  agent )  is a hyper-parameter for balancing the weights of different loss functions. In the inference phase, we also make some adjustments to beam search for our proposed method. We maintain two beams, one for user summary and one for agent summary. At each step of decoding, the k th sequence of the user summary beam should consider the states in the k th sequence of the agent summary beam for role attention. Once one beam has finished decoding, we keep the beam fixed and search for the other one. The beam search will finish if both beams have finished searching. There are two dialogue summarization datasets with role-oriented summarization tasks. Thus, we evaluate the effectiveness of our proposed method on both datasets. First, we experiment on a Chinese fine-grained customer service summarization dataset named CSDS 4 (Lin et al., 2021). It provides separate summaries for the user and the agent, and both may contain multiple topics. The other one is a Chinese medical inquiry summarization dataset MC 5 (Song et al., 2020). Each dialogue has a summary of the patient's description and a summary of the doctor's suggestion. We note them as user summary and agent summary as well. Most of the summaries in MC are extractive, and only a few are different from dialogue scripts. Moreover, most dialogues in MC have only one topic. Comparing two 4 https://github.com/xiaolinAndy/CSDS. 5 https://github.com/cuhksz-nlp/HET-MC. We use the official crawling script to acquire the dataset and divide some data from the training set as the validation set. Due to the website update, the data may have a slight difference compared with the version in the original paper. datasets, MC is easier to summarize while CSDS is more specific for role-oriented summarization and more challenging. The detailed statistics of the two datasets are given in Table 1. We apply the role interaction methods on two widely-used seq2seq models in the summarization community, including PGN (See et al., 2017) and BERTAbs (Liu and Lapata, 2019). Therefore, we will introduce these two backbone models and how we apply Role Interactions to them. PGN is an LSTM-based seq2seq model with a copy mechanism to copy words from the input and a coverage mechanism for constraining context attention. We set two PGN-based baselines for comparison. PGN-single is to separately train two PGN models for generating user summary and agent summary, while PGN-multi tries to jointly train two PGN models by sharing the same encoder. Both baselines adopt all the dialogue context as input. To apply role interactions, we choose the output of the LSTM cell in the decoder as the query to calculate cross attention and role attention. The attention object in role attention is the output of the LSTM cell from the other decoder. Since we calculate the cross attention for different roles separately, we use a learnable gate p role to control the weight of different cross attentions and add them together according to their weights to achieve the overall context attention distribution. It is also used for the copy and coverage mechanism. We set PGN-cross as adding cross attention interaction, PGN-self as adding decoder self-attention interaction, and PGN-both as adding both interactions. Transformer has been widely used in language understanding and generation models due to its strong representation ability and concurrency, especially in pretrained models (Devlin et al., 2019; Lewis et al., 2020). Here we choose BERTAbs (Liu and Lapata, 2019) as the backbone structure since it performs well on many summarization datasets and is available for non-English languages such as Chinese. It adopts a pretrained BERT model as encoder and a transformer decoder structure to decode summaries. Both the encoder and the decoder contain six layers, and each layer contains three sub-layers (self-attention, encoder-decoder attention, feedfor-ward). Similar with PGN-based methods, we set BERT-single and BERT-multi as two baselines. We apply both interactions to each layer in BERTAbs. For cross attention interaction, we change the encoder-decoder attention sub-layer into two separate cross attention modules; for decoder self-attention interaction, we add the role attention module parallel with the encoder-decoder attention module. The query, key, and value of the role attention module are all the output from the self-attention sub-layer. BERT-cross , BERT-self , and BERT-both are kept the same with the settings in PGN-based methods. We add the role information to the front of the utterance in each turn and concatenate all the utterances in the dialogue sequentially as the input of the model. Both PGN 6 and BERTAbs 7 baseline methods are adopted from publicly available codes. For PGN-based methods, we use pretrained Chinese word vectors provided by Tencent 8 , and the vocabulary size is 10,000. While for BERTAbs-based methods, we use the base version of Chinese BERT-wwm 9 . The best checkpoint is chosen based on validation set loss, and we use beam search to decode summaries on the best checkpoint with beam size 5. For choosing hyper-parameters, since the agent summary is more complex than the user summary in MC, we set  to be 0.2 to give the agent summary more weight. It is set to be 0.5 for CSDS. is set to be 0.5 for PGN and 0.25 for BERTAbs. The hyper-parameter settings are chosen by experimenting on the validation set. More details are given in Appendix A. 6 https://github.com/atulkum/pointer_summarizer 7 https://github.com/nlpyang/PreSumm 8 https://ai.tencent.com/ailab/nlp/en/embedding.html 9 https://github.com/ymcui/Chinese-BERT-wwm 4.3 Evaluation Metrics We adopt six common automatic evaluation metrics to evaluate the summary quality. The metrics include traditional n-gram overlapping metrics, such as ROUGE (Lin and Hovy, 2002), BLEU (Pap-ineni et al., 2002), and distributed representation matching metrics, including BERTScore (Zhang et al., 2020) and MoverScore (Zhao et al., 2019). We use files2rouge toolkit to calculate the F1 score of ROUGE-1, ROUGE-2, ROUGE-L. More details of evaluation scripts are given in Appendix A. In addition to automatic metrics, we also compare the summary quality at a fine-grained level through human evaluation. Following the human evaluation process in Lin et al. (2021), we recruit several volunteers and let them evaluate the summaries in the following aspects: (1) Informativeness : Does the generated summary correctly cover the information in the ground truth summary? (2) Non-redundancy : Does the generated summary not contain repeated, meaningless or unnecessary information? (3) Fluency : Is the generated summary well-formed, semantically complete, and easy to understand? All three aspects are evaluated at the sub-summary level 10 on a three-point scale, 0 for the worst, 1 for the medium, and 2 for the best. First, we present the results of automatic metrics with Student's t-test as significance test in Table 2 and 3. The results are similar on the two datasets. First, the multi-task mechanism could bring some improvement than separately training on most of the metrics. However, the improvement is limited, especially for the PGN model on CSDS. After adding the enhancement of other roles' information, the performance is significantly boosted. On CSDS, PGN-single and BERT-single are two strong baselines provided in Lin et al. (2021) 11 . For PGN-based methods, the best method PGN-both utilizes two interactions and achieves 2.84 and 1.53 higher points on ROUGE-L for user summary and agent summary than PGN-single. For BERTAbs-10 We split summaries into different topic segments, and each segment is a sub-summary, same as the process inLin et al. (2021). 11 Note that we do not mention the baseline Fast-RL (Chen and Bansal, 2018) in Lin et al. (2021). It first extracts salient utterances and then generates summary sentences from each utterance separately, which is not available to add our proposed interaction methods. based methods, the improvements are even greater, which are 4.73 and 2.69. We also conduct ablation studies by only applying one interaction ( -cross or -self ). Both settings show promising improvement over the single and multi baselines on nearly all the metrics, demonstrating the effectiveness of each interaction method. In comparison, applying two interactions together yields the best result on the majority of metrics. The circumstance is similar on MC. User summarization is relatively simple on MC, and the baseline methods could achieve high performance (5.35 points of ROUGE-2 higher than the best performance in the original paper (Song et al., 2020)). Despite this, both cross attention interaction and decoder self-attention interaction could still increase the performance of user summary a bit. Additionally, the improvement on agent summary is more significant. PGN-both method achieves 0.90 points of ROUGE-2 and 1.29 points of MoverScore improvement, while BERT-both achieves 0.76 points of ROUGE-2 and 0.66 points of MoverScore improvement. PGN-both also beats the best result in the original paper on most of the metrics, which uses additional information such as hospital department and disease name. In conclusion, our proposed two interaction methods could bring remarkable improvement on different backbone structures CSDS Info Non-Red Flu Overall PGN-multi 0.69 /0.65 0.54/0.55 0.70/0.79 0.64/0.66 PGN-both 0.66/ 0.69 0.58 / 0.59* 0.73 / 0.81 0.66 / 0.70* BERT-multi 0.58/0.56 0.66 / 0.61 0.84/ 0.87 0.69/0.68 BERT-both 0.62* / 0.60* 0.62/0.60 0.85 / 0.87 0.70 / 0.69 Table 4: The human evaluation results for CSDS. To evaluate the summary quality at a more fine-grained level, we compare the summaries from different models according to the pre-defined three aspects: informativeness, non-redundancy, fluency. Since the multi-task framework works better than the single baseline, we directly compare it with applying both interactions. As CSDS is more challenging for this task, we randomly select 100 samples from the test set and obtain the outputs of two baseline methods (PGN-multi and BERT-multi) and two interaction methods (PGN-both and BERT-both). We recruit three volunteers and train them on the evaluation rules 12 . Then we let them evaluate the generated summaries according to the ground 12 More details are in Appendix C with ethical concerns. truth and the original dialogue in the three aspects. We run the inter-annotator agreement study on three volunteers' scores, and obtain a reasonable kappa score, 0.48 on average. We also calculate an Overall metric by averaging the results of all three aspects to represent the summary quality in general. We normalize the result into 0 to 1 and present it in Table 4. The result shows different trends on two backbone structures. For the PGN model, applying interactions could largely reduce the redundancy of both user and agent summary, with a comparable performance of informativeness. Besides, it also improves the fluency of the two summaries. For the BERTAbs model, the interaction method significantly improves the informativeness while the redundancy also increases a bit. The difference exists because BERTAbs prefers to generate short summaries. Thus, considering information from other roles could help generate more useful information but also induce some redundant text. In contrast, PGN tends to generate lengthy summaries. When considering information from other roles, it first tries to discard the redundant texts and only retains more important ones. The fluency improvement on both methods proves that other roles' information helps generate more semantically complete summaries. Considering the overall metric, we conclude that our proposed interaction method is also effective through human evaluation. Agent Summary Completeness Analysis The agent summary often suffers semantic incompleteness due to missing key information from other roles (Lin et al., 2021). Since our proposed role interactions aim at extracting other roles' key information, we wonder whether they work on these incomplete cases. Following the settings in Lin et al. (2021), we compare the summary quality of samples that need to integrate other roles' information and those that do not need separately 13 . The result in Table 5 shows that the interaction method could actually help improve the performance on samples that need to integrate. Besides, samples that do not need also get improved. We believe that it is because considering other roles' information could also help extract critical content from the role's own utterances as well. Topic Structural Summary Analysis Since we assume that role interactions could help generate better summaries by sharing the same discussion topic, we wonder whether the summaries generated by our methods could include the correct topic structure. More specifically, we want to find out the performance of our methods on summarizing each topic. Following the evaluation method in (Lin et al., 2021), we treat each sentence in the summary as a sub-summary for a single topic and calculate the number of matching sub-summaries with the reference by a ROUGE-L-based matching algorithm. We calculate the precision, recall, and F1 scores of correctly matched sub-summary ratios and present them in Table 6. The result shows that two role interaction methods achieve higher recall and F1 scores on sub-summary matching. It proves that role interactions could help the model grasp the discussion topic in the dialogue and generate a more accurate summary for each topic. We also present an example in Appendix B to prove the effectiveness of our proposed role interaction method. In this paper, we focus on the role-oriented dialogue summarization task. To fully exploit the information from other roles, we propose two role interaction methods on cross attention and decoder 13 It is judged by considering whether the summary needs to refer to other roles' utterances, which is already labeled in CSDS. self-attention. The cross attention interaction calculates the context information for different roles separately and uses same-role attention to guide cross-role attention. The decoder self-attention interaction adds a role attention module to attend to other role decoder states interactively. Experiments on two dialogue summarization datasets prove that both interactions perform significantly better than strong baseline methods. Adding role interactions also helps generate summaries with complete semantics and correct topic structure. In the future, we will try to apply this method to other dialogue-related tasks and conduct more experiments on stronger summarization methods. We thank all volunteers for their great help on the evaluation, and all anonymous reviewers' suggestions are very appreciated. The research work described in this paper has been partially supported by the National Key Research and Development Program of China under Grant No. 2020AAA0108600 and the Natural Science Foundation of China under Grant No. 62106263.
abstain abstain method objective abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective objective objective result objective abstain result abstain method method abstain objective objective result result abstain other other other other other other other other other other other other other objective other other other objective objective abstain abstain method method abstain method method method abstain result abstain abstain method method result abstain abstain abstain abstain method abstain other abstain abstain method result abstain abstain abstain method abstain abstain abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain result abstain abstain method abstain abstain method abstain result abstain method abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain abstain method other other