Abstractive conversation summarization has received much attention recently. However, these generated summaries often suffer from insufficient, redundant, or incorrect content, largely due to the unstructured and complex characteristics of human-human interactions. To this end, we propose to explicitly model the rich structures in conversations for more precise and accurate conversation summarization, by first incorporating discourse relations between utterances and action triples ( WHODOING-WHAT ) in utterances through structured graphs to better encode conversations, and then designing a multi-granularity decoder to generate summaries by combining all levels of information. Experiments show that our proposed models outperform state-of-the-art methods and generalize well in other domains in terms of both automatic evaluations and human judgments. We have publicly released our code at https://github.com/ GT-SALT/Structure-Aware-BART . Online interaction has become an indispensable component of everyday life and people are increasingly using textual conversations to exchange ideas, make plans, and share information. However, it is time-consuming to recap and grasp all the core content within every complex conversation (Gao et al., 2020; Feng et al., 2020). As a result, how to organize massive everyday interactions into natural, concise, and informative text, i.e., abstractive conversation summarization, starts to gain importance. Significant progress has been made on abstractive summarization for structured document via pointer generator (See et al., 2017), reinforcement methods (Paulus et al., 2018; Huang et al., 2020a) and pre-trained models (Liu and Lapata, 2019; Lewis et al., 2020; Zhang et al., 2019). Despite the huge success, it is challenging to directly apply document models to summarize conversations, due to Figure 1: An example of discourse relation graph a set of inherent differences between conversations and documents (Gliwa et al., 2019). First, speaker interruptions like repetitions, false-starts, and hesitations are frequent in conversations (Sacks et al., 1978), and key information resides in different portions of a conversation. These unstructured properties pose challenges for models to focus on salient contents that are necessary for generating both abstractive and informative summaries. Second, there is more than one speaker in conversations and people interact with each other in different language styles (Zhu et al., 2020b). The complex interactions among multiple speakers make it harder for models to identify and associate speakers with correct actions so as to generate factual summaries. In order to summarize the unstructured and complex conversations, a growing body of research has been conducted, such as transferring document summarization methods to conversation settings (Shang et al., 2018; Gliwa et al., 2019), adopting hierarchical models (Zhao et al., 2019; Zhu et al., 2020b), or incorporating conversation structures like topic segmentation (Liu et al., 2019b; Li et al., 2019; Chen and Yang, 2020), dialogue acts (Goo and Chen, 2018), and conversation stages (Chen and Yang, 2020). However, current approaches still face challenges in terms of succinctness and faithfulness, as most prior studies (i) fail to explicitly model dependencies between utterances which can help identify salient portions of conversations (Bui et al., 2009), and (ii) lack structured representations (Huang et al., 2020a) to learn the associations between speakers, actions and events. We argue that these rich linguistic structures associated with conversations are key components towards generating abstractive and factual conversation summaries. To this end, we present a structure-aware sequence-to-sequence model, in which we equip abstractive conversation summarization models with rich conversation structures through two types of graphs: discourse relation graph and action graph . Discourse relation graphs are constructed based on dependency-based discourse relations (Kirschner et al., 2012; Stone et al., 2013; Asher et al., 2016; Qin et al., 2017) between intertwined utterances, where each Elementary Discourse Unit (EDU) is one single utterance and they are linked through 16 different types of relations (Asher et al., 2016). As shown in Figure 1(a), highly related utterances are linked based on discourse relations like Question Answer Pairs , Comment and Explanation . Explicitly modeling these utterances relations in conversations can aid models in recognizing key content for succinct and informative summarization. Action graphs are constructed as the  WHO-DOING-WHAT  triplets in conversations which express socially situated identities and activities (Gee, 2014). For instance, in Figure 1(b), the action graph provides explicit information between Simon , fetch , and tissues for the utterance it is Simon who will fetch the tissues , making models less likely to generate summaries with wrong references (e.g., Helen will fetch the tissues ). propose to utilize discourse relation graphs and action graphs to better encode conversations for conversation summarization. (2) We design structure-aware sequence-to-sequence models to combine these structured graphs and generate summaries with the help of a novel multi-granularity decoder. (3) We demonstrate the effectiveness of our proposed methods through experiments on a large-scale conversation summarization dataset, SAMSum (Gliwa et al., 2019). (4) We further show that our structure-aware models can generalize well in new domains such as debate summarization. Document Summarization Compared to extractive document summarization (Gupta and Lehal, 2010; Narayan et al., 2018; Liu and Lapata, 2019), abstractive document summarization is generally considered more challenging and has received more attention. Various methods have been designed to tackle abstractive document summarization like sequence-to-sequence models (Rush et al., 2015), pointer generators (See et al., 2017), reinforcement learning methods (Paulus et al., 2018; Huang et al., 2020a) and pre-trained models (Lewis et al., 2020; Zhang et al., 2019). To generate faithful abstractive document summaries (Maynez et al., 2020), graph-based models were introduced recently such as extracting entity types (Fernandes et al., 2018; Fan et al., 2019), leveraging knowledge graphs (Huang et al., 2020a; Zhu et al., 2020a) or designing extra fact correction modules (Dong et al., 2020). Inspired by these graph-based methods, we also construct action graphs for generating more factual conversation summaries. Conversation Summarization Extractive dialogue summarization (Murray et al., 2005) has been studied extensively via statistical machine learning methods such as skip-chain CRFs (Galley, 2006), SVM with LDA models (Wang and Cardie, 2013), and multi-sentence compression algorithms (Shang et al., 2018). Such methods struggled with generating succinct, fluent, and natural summaries, especially when the key information needs to be aggregated from multiple first-person point-of-view utterances (Song et al., 2020). Abstractive conversation summarization overcomes these issues by designing hierarchical models (Zhao et al., 2019; Zhu et al., 2020b), incorporating commonsense knowledge (Feng et al., 2020), or leveraging conversational structures like dialogue acts (Goo and Chen, Figure 2: Model architecture. 2018), key point sequences (Liu et al., 2019a), topic segments (Liu et al., 2019b; Li et al., 2019) and stage developments (Chen and Yang, 2020). Some recent research has also utilized discourse relations as input features in classifiers to detect important content in conversations (Murray et al., 2006; Bui et al., 2009; Qin et al., 2017). However, current models still have not explicitly utilized the dependencies between different utterances, making models hard to leverage long-range dependencies and utilize these salient utterances. Moreover, less attention has been paid to identify the actions of different speakers and how they interact with or refer to each other, leading to unfaithful summarization with incorrect references or wrong reasoning (Gliwa et al., 2019). To fill these gaps, we propose to explicitly model actions within utterances, and relations between utterances in conversations in a structured way, by using discourse relation graphs and action graphs and further combining these through relational graph encoders and multi-granularity decoders for abstractive conversation summarization. To generate abstractive and factual summaries from unstructured conversations, we propose to model structural signals in conversations by first constructing discourse relation graphs and action graphs (Section 3.1), and then encoding the graphs together with conversations (Section 3.2) as well as incorporating these different levels of information in the decoding stage through a multi-granularity decoder (Section 3.3) to summarize given conversations. The overall architecture is shown in Figure 2. 3.1 Structured Graph Construction This section describes how to construct the discourse relation graphs and action graphs. Formally, for a given conversation C = { u 0 , ..., u m } with m utterances, we construct discourse relation graph GD = ( VD , ED ) , where VD is the set of nodes representing Elementary Discourse Units (EDUs), and ED is the adjacent matrix that describes the relations between EDUs, and action graph GA = ( VA , EA ) , where VA is the set of nodes representing  WHO ,  DOING  and  WHAT  arguments, and EA is the adjacent matrix to link  WHO-DOING-WHAT  triples. Discourse Relation Graph Utterances from different speakers do not occur in isolation; instead, they are related within the context of discourse (Murray et al., 2006; Qin et al., 2017), which has been shown effective for dialogue understanding like identifying the decisions in multi-party dialogues (Bui et al., 2009) and detecting salient content in email conversations (McKeown et al., 2007). Although current attention-based neural models are supposed to, or might implicitly, learn certain relations between utterances, they often struggle to focus on many informative utterances (Chen and Yang, 2020; Song et al., 2020) and fail to address long-range dependencies (Xu et al., 2020), especially when there are frequent interruptions. As a result, explicitly incorporating the discourse relations will help neural summarization models better encode the unstructured conversations and concentrate on the most salient utterances to generate more informative and less redundant summaries. To do so, we view each utterance as an EDU and use the discourse relation types defined in Asher et al. (2016). We first pre-train a discourse parsing model (Shi and Huang, 2019) on a human-annotated multiparty dialogue corpus (Asher et al., 2016), with 0.775 F1 score on link predictions and 0.557 F1 score on relation classifications, which are comparable to the state-of-the-art results (Shi and Huang, 2019). We then utilize this pre-trained parser to predict the discourse relations within conversations in our SAMSum corpus (Gliwa et al., 2019). After predictions, there are 138,554 edges iden-tified in total and 8.48 edges per conversation. The distribution of these predicted discourse relation types is: Comment (19.3%), Clarification Question (15.2%), Elaboration (2.3%), Acknowledge-ment(8.4%), Continuation (10.1%), Explanation (2.8%), Conditional (0.2 %), Question Answer Pair (21.5%), Alternation (0.3%), Q-Elab (2.5%), Result (5.5%), Background (0.4%), Narration (0.4%), Correction (0.4%), Parallel (0.9%), and Contrast (1.0%). Then for each conversation, we construct a discourse relation graph GD = ( VD , ED ) , where VD [ k ] represents the k -th utterance. ED [ i ][ j ] = r if there is a link from the i -th utterance to the j -th one with discourse relation r . Action Graph The  who-doing-what  triples from utterances can provide explicit visualizations of speakers and their actions, the key to understanding concrete details happened in conversations (Moser, 2001; Gee, 2014; Sacks et al., 1978). Simply relying on neural models to identify this information from conversations often fail to produce factual characterizations of concrete details happened (Cao et al., 2018; Huang et al., 2020a). To this end, we extract  WHO-DOING-WHAT  triples from utterances and construct action graphs for conversation summarization (Chen et al., 2019; Huang et al., 2020b,a). Specifically, we first transform the first-person point-of-view utterances to its third-person point-of-view forms based on simple rules: (i) substituting first/second-person pronouns with the names of current speaker or surrounding speakers and (ii) replacing third-person pronouns based on coreference clusters in conversations detected by the Stanford CoreNLP (Manning et al., 2014). For example, an utterance  I'll bring it to you tomorrow  from Amanda to Jerry will be transformed into  Amanda'll bring cakes to Jerry tomorrow . Then we extract  WHO-DOING-WHAT  (subject-predicate-object) triples from transformed conversations using the open information extraction (Ope-nIE) systems 1 (Angeli et al., 2015). We then construct the Action Graph GA = ( VA , EA ) from the extracted triples by taking arguments ( WHO ,  DOING , or  WHAT  ) as nodes in VA , and connect them with edge EA [ i ][ j ] = 1 if they are adjacent in one  WHO-DOING-WHAT  triple. Given a conversation and its corresponding discourse relation graph and action graph, we utilize an utterance encoder and two graph encoders, to obtain its hidden representations shown in Figure 2(a). We initialize our utterance encoder FU ( . ) with a pre-trained encoder, i.e., BART-base (Lewis et al., 2020), and encode tokens { x i, 0 , ..., x i,l } in an utterance u i into its hidden representation: Node Initialization For discourse relation graph , we employ the output embeddings of the special tokens x i, 0 from the utterance encoder, i.e., h Ui, 0 , to initialize the i -th node v Di in GD . We use a one-hot embedding layer to encode the relations ED [ i ][ j ] = e Di,j between utterance i and j . For action graph , we first utilize FU ( . ) to encode each token in nodes v Ai and then average their output embeddings as their initial representations. Structured Graph Attention Network Based on Graph Attention Network (Velickovic et al., 2018), we utilize these relations between nodes to encode each node v Di in GD or v Ai in GA through:  ij = exp (cid:0)  (cid:0) a T [ W v i (cid:107) W v j (cid:107) W e e i,j ] (cid:1)(cid:1) (cid:80) k N i exp (  ( a T [ W v i (cid:107) W v k (cid:107) W e e i,k ])) h i =  ( (cid:88) j N i  ij W v j ) 1 https://github.com/philipperemy/ Stanford-OpenIE-Python Dataset Split # Conv # Participants # Turns # Discourse Edges # Action Triples SAMSum Train 14732 2.40 11.17 8.47 6.72 Val 818 2.39 10.83 8.34 6.48 Test 819 2.36 11.25 8.63 6.81 ADSC Full 45 2.00 7.51 6.51 37.20 Table 1: Statistics of the used datasets, including the total number of conversations (# Conv), the average number of participants, turns, discourse edges and action triples per conversation. W , W e and a are trainable parameters. [. (cid:107) .] denotes the concatenation of two vectors. is the activation function, N i is the set containing node-i 's neighbours in G . Different levels of encoded representations are then aggregated via our multi-granularity decoder to generate summaries as shown in Figure 2(b). With s  1 previously generated tokens y 1 , ..., y s  1 , our decoder G ( . ) predicts the l -th token via:  y = G ( y 1: s  1 , FU ( C ) , FD ( GD ) , FA ( GA )) (4) P ( y s | y <s , C , GD , GA ) = Softmax ( W p  y ) (5) To better incorporate the information in constructed graphs, different from the traditional pretrained BART model (Lewis et al., 2020), we improve the BART transformer decoder with two extra cross attentions (Discourse Attention and Action Attention) added to each decoder layer, which attends to the encoded node representations in discourse relation graphs and action graphs. In each decoder layer, after performing the original cross attentions over every token in utterances { h Ui, 0: l } and getting the utterance-attended representation x U , multi-granularity decoder then conducts cross attentions over nodes { h D 0: m } and { h A 0: n } that are encoded from graph encoders in parallel, to obtain the discourse-attended representation x D and action-attended representation x A . These two attended vectors are then combined into a structure-aware representation x S , through a feed-forward network for further forward passing in the decoder. To alleviate the negative impact of randomly initialized graph encoders and cross attentions over graphs on pre-trained BART decoders at early stages and accelerate the learning of newly-introduced modules during training, we apply ReZero (Bachlechner et al., 2020) to the residual connection after attending to graphs in each decoder layer:  x S = x U + x S (6) where  is one trainable parameter instead of a fixed value 1, which modulates updates from cross attentions over graphs. Training During training, we seek to minimize the cross entropy and use the teacher-forcing strategy (Bengio et al., 2015): L =  (cid:88) log P ( y l | y <l , C , GD , GA ) (7) 4 Experiments 4.1 Datasets We trained and evaluated our models on a conversation summarization dataset SAMSum (Gliwa et al., 2019) covering messenger-like conversations about daily topics, such as arranging meetings and discussing events. We also showed the generalizability of our models on the Argumentative Dialogue Summary Corpus (ADSC) (Misra et al., 2015), a debate summarization corpus. The data statistics of two datasets were shown in Table 1, with the discourse relation types distributions in the Appendix. Pointer Generator (See et al., 2017): We followed the settings in Gliwa et al. (2019) and used special tokens to separate each utterance. Transformer (Vaswani et al., 2017): We trained transformer seq2seq models following the OpenNMT (Klein et al., 2017). D-HGN (Feng et al., 2020) incorporated commonsense knowledge from ConceptNet (Liu and Singh, 2004) for dialogue summarization. BART (Lewis et al., 2020): We utilized BART 2 , and separated utterances by a special token. Multi-View Seq2Seq (Chen and Yang, 2020) utilized topic and stage views on top of BART for summarizing conversations. Here we implemented it based on BART-base models. We used the BART-base model to initialize our sequence-to-sequence model for training in all experiments. For parameters in the original BART encoder/decoder, we followed the default settings and set the learning rate 3e-5 with 120 warm-up steps. For graph encoders, we set the number of hidden dimensions as 768, the number of attention heads as 2, the number of layers as 2, and the dropout rate as 0.2. For graph cross attentions added to BART decoder layers, we set the number of attention heads as 2. The weights  in ReZero residual connections were initialized with 1. The learning rate for parameters in newly added modules was 3e-4 with 60 warm-up steps. All experiments were performed on GeForce RTX 2080Ti (11GB memory). ROUGE scores (Lin and Och, 2004) 3 , and reported ROUGE-1, ROUGE-2, and ROUGE-L in Table 2. We found that, compared to simple sequence-to-sequence models ( Pointer Generator and Transformer ), incorporating extra information such as commonsense knowledge from ConceptNet ( D-HGN ) increased the ROUGE metrics. When equipped with pre-trained models and simple conversation structures such as topics and conversation stages, Multi-View Seq2Seq boosted ROUGE scores. Incorporating discourse relation graphs or action graphs helped the performances of summarization, suggesting the effectiveness of explicitly modeling relations between utterances and the associations between speakers and actions within utterances. Combining two different structured graphs produced better ROUGE scores compared to previous state-of-the-art methods and our base models, 3 We followed fairseq and used https://github. com/pltrdy/rouge to calculate ROUGE scores. Note that different tools may result in different ROUGE scores. with an increase of 2.0% on ROUGE-1, 4.3% on ROUGE-2, and 1.2% on ROUGE-L compared to our base model, BART . This indicates that, our structure-aware models with discourse and action graphs could help abstractive conversation summarization, and these two graphs complemented each other in generating better summaries. Human Evaluation We conducted human evaluation to qualitatively evaluate the generated summaries. Specifically, we asked annotators from Amazon Mechanical Turk to score a set of randomly sampled 100 generated summaries from ground-truth, BART and our structured models, using a Likert scale from 1 (worst) to 5 (best) in terms of factualness (e.g., associates actions with the right actors) , succinctness (e.g., does not contain redundant information), and informativeness (e.g., covers the most important content) (Feng et al., 2020; Huang et al., 2020a). To increase annotation quality, we required turkers to have a 98% approval rate and at least 10,000 approved tasks for their previous work. Each message was rated by three workers. The scores for each summary were averaged. The Intra-Class Correlation was 0.543, showing moderate agreement (Koo and Li, 2016). As shown in Table 4, S-BART that utilized structured information from discourse relation graphs and action graphs generated significantly better summaries with respect to factualness, succinctness, and informativeness. This might because that the incorporation of structured information such as discourse relations helped S-BART to recognize the salient parts in conversations, and thus improve the succinctness and informativeness over BART . Modeling the connections between speakers and actions greatly helped generate more factual summaries than the baselines, e.g., with an increase of 0.27 from BART to S-BART w. Action . To investigate the generalizability of our structure-aware models, we then tested the S-BART model trained on SAMSum corpus directly on the debate summarization domain (ADSC Corpus (Misra et al., 2015)) in a zero-shot setting. Besides the differences in topics, utterances in debate conversations were generally longer and include more action triples (37.20 vs 6.81 as shown in Table 1) and fewer participants. The distribution of discourse relation types also differed a lot across different Graph Types R-1 R-2 R-LS-BART w. As shown in Table 3, our single graph models S-BART w. Discourse and S-BART w. Action boosted ROUGE scores compared to BART , suggesting that utilizing structures can also increase the generalizability of conversation summarization methods. However, contrary to in-domain results in Table 2, action graphs led to much more gains than discourse graphs. This indicated that when domain shifts, action triples were most robust in terms of zero-shot setups; differences in discourse relation distributions could limit such generalization. Consistent with in-domain scenarios, our S-BART w. Discourse&Action achieved better results, with an increase of 66.2% on ROUGE-1, 373.4% on ROUGE-2, and 82.2% on ROUGE-L over BART . The Quality of Discourse Relation Graphs We showed how the quality of discourse relation graphs affected the performances of conversation summarization in Table 5. Specifically, we compared the ROUGE scores of S-BART using our constructed discourse relation graphs ( S-BART w. Discourse Graph ) and S-BART using randomly generated discourse relation graphs S-BART w. Random Graph where both connections between nodes and relation types were randomized. The number of edges in two graphs was kept the same. We found that S-BART with our discourse graphs outperformed 4 The detailed distributions were shown in the Appendix. Different Ways to Combine Graphs We experimented with different ways to combine discourse relation graphs and action graphs in our S-BART w. Discourse & Action , and presented the results in Table 6. Here, parallel strategy performed cross attentions on different graphs separately and then combined the attended results with feed-forward networks as discussed in Section 3.3; sequential strategy performed cross attentions on two graphs in a specific order (from discourse relation graphs to actions graphs, or vice versa). We found that the parallel strategy showed better performances and the sequential ones did not introduce gains compared to S-BART with single graphs. This demonstrates that discourse relation graphs and action graphs were both important and provided different signals for abstractive conversation summarization. Visualizing ReZero Weights We further tested our structure-aware BART with two ReZero settings: (i) initializing  from 0, (ii) initializing  from 1, and found initializing  from 1 would bring in more performance gains (see Appendix). We then visualized the average  over different decoder layers after training in Figure 3, and observed that (i) when  was initialized with 1, the final  was much larger than the setting where  was initialized with 0, which might because randomly initialized modules barely received supervisions at early stages and therefore contributes less to BART . (ii) Compared to discourse graphs, action graphs received higher  weights after training in both initializing settings, suggesting that the information from structured action graphs might be harder for the end-to-end BART models to capture. (iii) Utilizing both graphs spontaneously led to higher Conversations # Num # Dis. ReZero weights, further validating the effectiveness of combining discourse relation graphs and action graphs and their complementary properties. To inspect when our summarization models could help the conversations summarization, we visualized the average number of discourse edges and the average number of action triples in three sets of conversations in Table 7: (i) Similar : examples where S-BART generated similar ROUGE scores (the differences were less than 0.1) compared to BART ; (ii) Increase : examples where S-BART resulted in higher ROUGE scores (the differences were larger than 1.0) compared to BART ; (iii) Challenging : examples where both S-BART and BART showed low ROUGE scores (ROUGE-1 < 20.0, ROUGE-2 < 10.0, ROUGE-L < 10.0). When the structures in conversations were simpler (fewer discourse edges and fewer action triples than the average), BART showed similar performance as S-BART . As the structures of conversations become more complex with more discourse relations and more action mentions, S-BART outperformed BART as it explicitly incorporated these structured graphs. However, both BART and S-BART struggled when there were much more interactions beyond certain thresholds, calling for better mechanisms to model structures in conversations for generating better summaries. In this work, we introduced a structure-aware sequence-to-sequence model for abstractive conversation summarization by incorporating discourse relations between utterances, and the connections between speakers and actions within utterances. Experiments and ablation studies on SAMSum corpus showed the effectiveness of these structured graphs in aiding the task of conversation summarization via both quantitative and qualitative evaluation metrics. Results in zero-shot settings on ADCS Corpus further demonstrated the generalizability of our structure-aware models. In the future, we plan to extend our current conversation summarization models for various application domains such as emails, debates, and podcasts, and in conversations that might involve longer utterances and more participants in an unsynchronized way. We would like to thank the anonymous reviewers for their helpful comments, and the members of Georgia Tech SALT group for their feedback. This work is supported in part by grants from Google, Amazon and Salesforce.
abstain abstain objective objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective other other other method other other other other other other other objective method abstain abstain method other abstain abstain method method method abstain other method abstain abstain abstain method method abstain abstain abstain method method method abstain method method method method other abstain abstain abstain method abstain method other abstain method method method abstain abstain method abstain method abstain method method method abstain method abstain result abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective other other