Although current state-of-the-art Transformer-based solutions succeeded in a wide range for single-document NLP tasks, they still struggle to address multi-input tasks such as multi-document summarization. Many solutions truncate the inputs, thus ignoring potential summary-relevant contents, which is unacceptable in the medical domain where each information can be vital. Others leverage linear model approximations to apply multi-input concatenation, worsening the results because all information is considered, even if it is conflicting or noisy with respect to a shared background. Despite the importance and social impact of medicine, there are no ad-hoc solutions for multi-document summarization. For this reason, we propose a novel discriminative marginalized probabilistic method (DAMEN) trained to discriminate critical information from a cluster of topic-related medical documents and generate a multi-document summary via token probability marginalization. Results prove we outperform the previous state-of-the-art on a biomedical dataset for multi-document summarization of systematic literature reviews. Moreover, we perform extensive ablation studies to motivate the design choices and prove the importance of each module of our method. 1 1 Introduction The task of multi-document summarization aims to generate a compact and informative summary from a cluster of topic-related documents, which represents a very challenging natural language processing (NLP) application due to the presence of redundant and sometimes conflicting information among documents (Radev, 2000). In the medical domain, in which machine learning plays an increasingly significant role (Domeniconi et al., 2014a; di Lena et al., 2015), multi-document summarization finds application in the generation of 1 The solution of this paper is available at https:// disi-unibo-nlp.github.io/projects/damen systematic literature reviews, a biomedical paper that summarizes results across many studies (Khan et al., 2003). DeYoung et al. (2021) are the first that address this task, showing the related issues. State-of-the-art approaches leverage two leading solutions: hierarchical networks that capture cross-document relations via graph encodings (Wan and Yang, 2006; Liao et al., 2018; Li et al., 2020; Pa-sunuru et al., 2021) or hidden states aggregation (Fabbri et al., 2019; Liu and Lapata, 2019a; Jin et al., 2020), and long-range neural models that apply multi-input concatenation (Xiao et al., 2021). While effective, these solutions struggle to process clusters of many topic-related documents in low computational resource scenarios (Moro and Ragazzi, 2022) because they need to truncate the inputs. Moreover, pre-trained state-of-the-art Transformers are not leveraged despite showing strong performance when fine-tuned in downstream tasks such as single-document summarization (Liu and Lapata, 2019b; Lewis et al., 2020a; Raffel et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020). Multi-document summarization requires models to have more robust capabilities for analyzing the cluster to discriminate the correct information from noise and merge it consistently. In this work, we propose a discriminative marginalized probabilistic neural method (DAMEN) that selects worthy documents in the cluster with respect to a shared background and generates the summary via token probability marginalization. The marginalization of the probability has been successfully applied in past NLP models such as pLSA (Hofmann, 1999) to learn the word probability distribution in documents by maximizing the likelihood. Recently, new deep neural models that use probability marginalization approaches have been proposed for the question-answering task, where, however, each input/output sentence is generally several orders of magnitude shorter than sets of documents in multi-document summarization 180 (Guu et al., 2020; Lewis et al., 2020b). To the best of our knowledge, we are the first that propose such a method for multi-document summarization. To this aim, we conduct experiments on the only medical dataset for multi-document summarization of systematic literature reviews (MS2). Besides, we perform extensive ablation studies to motivate the design choices and prove the importance of each component of our method. To sum up, our contributions are as follows:  We propose a novel probabilistic neural method for multi-document summarization (DAMEN) that discriminates the summary-relevant information from a cluster of topic-related documents and generates a final summary via token probability marginalization. We advance the research in the medical domain, experimenting with a biomedical multi-document summarization dataset about the generation of systematic literature reviews. We show that our solution outperforms previous state-of-the-art solutions, achieving better ROUGE scores. Furthermore, we extensively prove the contribution of each module of our method with ablation studies. We describe related works on multi-document summarization categorized on model architectures. Flat solutions. Flat concatenation is a simple yet powerful solution because the generation of the multi-document summary is treated as a single-document summarization task, thus it can leverage state-of-the-art pre-trained summarization models. Consequently, processing all documents as a flat input requires models capable of handling long sequences. As previously experimented by DeYoung et al. (2021), Xiao et al. (2021) proposed to leverage the Longformer-Encoder-Decoder model (Beltagy et al., 2020) pre-trained with a novel multi-document summarization specific task. They proved that a long-range Transformer that encodes all documents is a straightforward yet effective solution, and they achieved new state-of-the-art results in several multi-document summarization datasets. However, such models may struggle to handle a massive cluster of topic-related documents since they need to truncate them because of architectural limits. Further, processing all documents in a cluster could be noisy if some of them are not relevant or factual with respect to the summary. Hierarchical solutions. To better preserve cross-document relations and obtain semantic-rich representations, hierarchical concatenation solutions leverage graph-based techniques to work from word and sentence-level (Wan and Yang, 2006; Liao et al., 2018; Nayeem et al., 2018; Antognini and Faltings, 2019; Li et al., 2020) to document-level (Amplayo and Lapata, 2021). Other hierarchical approaches include multi-head pooling and inter-paragraph attention architectures (Liu and Lapata, 2019a), attention models with maximal marginal relevance (Fabbri et al., 2019), and attention across different granularity representations (Jin et al., 2020). Such models are often dataset-specific because of the custom architecture, so they struggle to adapt to other datasets and effectively leverage pre-trained state-of-the-art Transformers. Our solution. In this work, we show how the summary-relevant information can be discriminated from a cluster of medical documents by a probabilistic neural method trained end-to-end. In detail, our solution fully leverages pre-trained state-of-the-art Transformers without applying input truncation that causes performance drop and discards important contents, unacceptable for a high-social impact domain such as the medical one. We introduce DAMEN, a discriminative marginalized probabilistic neural method for the multi-document summarization of medical literature based on three components: Indexer : it is a neural language model based on BERT architecture (Cohan et al., 2020) that creates a dense representation of documents in the cluster, according to the best practices for information retrieval systems. Discriminator : it leverages a BERT model to create the background embedding, which is used to compute a distance score between the embedding of each document in the cluster in order to select the top K ones. Generator : it uses a BART model (Lewis et al., 2020a) to produce the final summary via token probability marginalization from the top K documents combined with the background. While the Indexer is a frozen pre-trained model based on BERT, the Discriminator and Generator are trained end-to-end during the learning phase (Fig. 1). The overall task can be mathematically formalized as follows. The training tuple is composed of three elements ( y i , x i , C i ) , where y i is the ground-truth target summary, C i is the cluster of documents used to generate the multi-document summary, and x i is the background, which is a textual context shared by all c j  C i used as input of the method, similar to the query in the query-focused multi-document summarization (Su et al., 2020). The whole pipeline is trained end-to-end to maximize the conditional probability of generating y i from x i and C i through gradient descend: p ( y i | x i , C i ) (1) 3.1 Indexer In this phase, we index each document in the cluster with an embedding generated by a BERT-based model. Such a pre-trained language model is the state-of-the-art in semantic modeling from textual data thanks to the vast knowledge learned during pre-training (Chen et al., 2019), achieving ground-breaking results across an extensive range of NLP downstream tasks even without fine-tuning. For this reason, we use it to create a dense latent representation of each document, called document embedding, which is a vector of continuous numbers that indicates a point in a latent semantic space. The technique we use is known as dense passage retriever (DPR) (Karpukhin et al., 2020), and it is widely adopted in the information retrieval domain (e.g., Lin et al., 2021; Moro and Valgimigli, 2021). We choose the DPR method because it does not interrupt the backpropagation, differently from other solutions, e.g., BM25, TF-IDF (Domeniconi et al., 2014b) or LSA (Domeniconi et al., 2016a). We formalize this step as B  ( C i ) = E , where B is a BERT-based model,  represents its parameters, and E is a matrix of shape ( len ( C i ) , 768) , where each row j of the matrix is the latent representation of the document c j . The main idea of the Discriminator is to discriminate the critical information from noise in a cluster of topic-related documents with respect to a shared background without breaking the backpropagation chain. For this reason, we use a probabilistic deep neural model to draw a probability distribution over documents in the cluster < c 0 , c 1 , ..., c n >  C i , with the following formula: p  ( C i | x i ) (2) where  represents the parameters of the neural network. a BERT-based pre-trained language model as the one used for indexing, but this is trained during the learning process while the first is frozen. In detail, the Discriminator creates a latent projection for each background, which is used to fetch the more related documents in the cluster. More precisely, it applies the inner product to create a score for each document and selects the top K ones. We use the pre-trained encoder-decoder generative Transformer BART (Lewis et al., 2020a) to summarize the C i weighted by the Discriminator . This component is trained to predict the next output token, creating a probability distribution over the dictionary for each c j  C i before marginalizing. The process is then repeated for all the target tokens. Before giving the documents to the model, we concatenate them with the background x i , creating c  ij = [ x i , tok, c ij ] , where tok is a special text separator token ( <doc> ) we add between x i and c ij to make BART aware of the background text boundary. The behavior of the Generator can be formally defined as follows: p ( y i | c  ij ) = N (cid:89) z p  ( y iz | c  ij , y i, 1: z  1 ) (3) where  are the Generator parameters, N = | y i | is the target length, and y i, 1: z are the tokens from position 1 to z of the target y i . The entire model aims to draw the probability distribution over the dictionary to generate the output tokens y i conditioned by x i and C i that we formally define as: the following loss: This section starts with describing the dataset in 4.1 and training details in 4.2. We then analyze model performance in 4.3 and finally conduct ablation studies in 4.4. We tested and evaluated our proposed method on the only medical dataset for multi-document summarization, as far as we know, about the generation of systematic literature reviews: the MS2 dataset. The dataset is provided in DeYoung et al. (2021), and it is freely distributed. 2 It contains over 470K document abstracts and 20K summaries derived from the scientific literature. Each sample of the dataset is composed of three elements: i) the background statement, which is a short text that describes the research question or topic shared by all documents in the cluster, ii) the target statement, which is the multi-document summary to generate, and iii) the studies , also defined as cluster for consistency with our notation, which is a set of abstracts of medical studies related to the topic covered in the background statement. The problem can be formalized as follows: we have a target statement to generate about the background source, containing the topic specifications, and a cluster of related document abstracts from which to fetch and discriminate helpful knowledge with respect to the background. From here on, we use the terms document and abstract interchangeably since the elements in the cluster are just the abstracts of medical documents. We report the dataset statistics in Table 1. We trained our solution for 3 epochs using a batch size of 1 and a learning rate with a linear schedule set to 1  10  5 . We set the number of K equal to 6 because it gave best results and used 1024 tokens as the max input size for the Generator . During the evaluation, we adopted a beam size of 4 with a min and a max length set to 32 and 256, respectively. We implemented the code using PyTorch for tensor computations and Hugging Face 3 for language model checkpoints. We performed the experiments on a workstation with a GPU Nvidia RTX 3090 of 24GB memory, 64GB of RAM, and a processor Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz. Table 2 shows the results on multi-document summarization of systematic literature reviews, comparing our method with two solutions proposed in DeYoung et al. (2021). The BARTHIERARCHICAL solution is trained to encode each document independently and then concatenate the representation of hidden states before decoding, whereas LEDFLAT takes as input all documents concatenated as a single document. Experimental results show we outperform the state-of-the-art in all the ROUGE metrics, proving better capability to discriminate relevant information across many related documents and merge it consistently (Fig. 2). We conducted ablation studies on the MS2 dataset to prove the importance of each module of our method. In detail, for all experiments we trained our solution for 1 epoch with the same training details reported in 4.2, and we performed the evaluation on the first 400 instances of the test set. The importance of a highly abstractive large-sized Generator. We report in Table 3 the performance using several pre-trained checkpoints of the Generator that differ in size and training. In detail, we tested two BARTBASE checkpoints and three BARTLARGE checkpoints:  facebook/bart-base : the actual BART model pre-trained with a denoising masked language modeling. gayanin/bart-mlm-pubmed : the BART model pre-trained exclusively on scientific corpora. facebook/bart-large : the same BART model as the base version with a large architecture. facebook/bart-large-cnn : the large BART fine-tuned on single-document summarization on the CNN/DailyMail dataset (Nallapati et al., 2016). facebook/bart-large-xsum : the large BART fine-tuned on single-document summarization on the XSum dataset (Narayan et al., 2018). Results prove that a large-sized BART model already fine-tuned on a summarization task achieves better performance. More precisely, the checkpoint fine-tuned on the XSum dataset obtains better results thanks to the higher abstractiveness and the shortness of the target summaries, which are made up of just 1-2 sentences, similar to the MS2 dataset. The importance of a full-sized chunked representation of documents in the cluster. Table 4 reports experiments with three cluster configurations, where each document is treated with a different text representation, described as follows:  Document-level : the simpler configuration that considers the entire abstracts in the cluster. We truncated documents taking only the first 512 tokens before encoding by the Indexer . Sentence-level : we considered the sentences of each document obtained using the state-of-the-art tokenizer PySBD (Sadvilkar and Neumann, 2020). The sentences are encoded up to 128 tokens in length and they are then treated as individual textual units. Chunk-level : our configuration, where each document is split into chunks of exact 512 tokens to consider all text information without input truncation. This configuration is similar to the sentence-level one but with the difference that each textual unit is 512 tokens in length and not 128. The results prove the better performance on a cluster with chunked documents. By considering 512 tokens for each document, we fully leverage the capability of BERT language modeling without truncating any information. Input truncation required by the document-level configuration plays an important role in final accuracy because it discards and ignores potential summary-relevant information, leading to a performance drop. The sentence-level setting lets us increase the top K sentences to retrieve, but it worsens the final summary because single sentences are too fine-grained. The importance of a background-first concatenation with special token. Table 5 reports the experiments with a different configuration of the concatenated inputs to give to the Generator . We experimented with four types of concatenation:  [Document + Background]  [Background + Document]  [Document + <doc> + Background]  [Background + <doc> + Document] Results prove the importance of a background-first concatenation with the special token separator to make BART aware of the textual difference between the background and the documents. model checkpoints for the Indexer and Discriminator . First, we leveraged the checkpoint  sentence-transformers/allenai-specter  (Cohan et al., 2020), which is a scientific BERT-based model trained to create document embeddings by using paper citations. Thus, we used this pre-trained model for both the Indexer and Discriminator . Second, we used two different checkpoints with a specific DPR training, such as  facebook/dpr-question_encoder-single-nq-base  for encoding the background and  facebook/dpr-ctx_encoder-single-nq-base  for encoding each document in the cluster. Results prove the importance of the DPR checkpoints for both the Indexer and Discriminator . We proposed a novel probabilistic method based on the combination of three language models to tackle multi-document summarization in the medical domain. This task is characterized by redundant information, noise, and the possible presence of vital information in each sentence that makes arbitrary input truncation unacceptable. For this reason, we proposed a multi-document summarization method able to discriminate salient contents from irrelevant before summarizing. In detail, the solution first leverages a BERT-based model ( Indexer ) for creating dense indices for each chunk of each document in the cluster. Then, a second BERT-based model ( Discriminator ) is used to process the shared background and select only the most relevant chunks. The final BART model is trained to perform a probability marginalization over each token prediction for each selected chunk. In this way, our solution reads all document information and selects just the most relevant chunks, discarding noise before feeding the Generator . The Discriminator and Generator are trained end-to-end, backpropagat-ing the probability distribution as explained in 3. The Indexer is frozen; training would lead to some MS2 Encoders Rouge-1 Rouge-2 Rouge-L w/o ad-hoc encoders SPECTER-based 27.79 8.60 21.23 w/ ad-hoc encoders DPR-based 28.35 8.96 21.62 Table 6: The ablations to validate the contribution of the Indexer and Discriminator model checkpoints. problems, such as the time to learn improved embeddings at each iteration and the larger memory occupation to save the gradient for each document. We tested our method on MS2, the only dataset on systematic literature reviews, and compared it with state-of-the-art models, finding that our novel approach outperforms competitors on the ROUGE evaluation metrics. Further, we performed extensive ablation studies to highlight the contribution of each component and motivate the design choices. At the edge of our knowledge, this is the first work that applies a probability marginalization method for multi-document summarization. We believe this work can inspire novel research towards end-to-end multi-model collaboration instead of solutions with a single large model addressing the entire task. According to the divide et impera pattern, each model learns a specific sub-task, creating a more efficient and transparent cooperating solution. Tasks such as related work generation or text generation from multi-sourced inputs can get the most from our method, improving pre-existing solutions to discriminate helpful knowledge from noise. Further possible directions to deal with multi-inputs are the following: i) extracting relevant snippets from documents with term weighting techniques (Domeniconi et al., 2015) or semantic relations with unsupervised methods (Domeniconi et al., 2016b, 2017) to better model interpretable representations based on knowledge graph learning techniques (Frisoni and Moro, 2020; Chen et al., 2021a,b) or event extraction methods (Frisoni et al., 2021); ii) training models to write and read cross-document information with self-supervised representation learning methods (Domeniconi et al., 2014c) and memory-based neural layers (Moro et al., 2018; Cui and Hu, 2021). The advancement of deep neural network architectures and the availability of large pre-trained language models has led to significant improvements for the multi-document summarization task, which has applications in high-impact domains, particularly in the medical one. Here, systematic literature reviews play an essential role for the medical and scientific community, and for that reason, they require strong guarantees about the factuality of the output summary. Current state-of-the-art NLP solutions cannot establish such assurance, so we do not believe our solution, like previous ones, is ready to be deployed. The research should explore more effective evaluation measures for text summarization to make it happen, and large-scale accuracy guarantees by medical experts are still needed. Finally, if the method will be applied to sensitive data such as medical patient records, it should also include privacy-preserving policies (da Silva et al., 2006). We thank the Maggioli Group 4 for granting the Ph.D. scholarship to L. Ragazzi and L. Valgimigli. The solution presented in this work has been designed by G. Moro.
abstain abstain abstain abstain objective result objective abstain other abstain other abstain abstain abstain objective abstain abstain objective objective objective objective method result objective objective other other other other other other other other other other other abstain abstain method method abstain abstain abstain abstain abstain other method abstain method method method method abstain method abstain abstain abstain method abstain abstain method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain objective abstain objective abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other