We introduce PRIMERA , a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins. 1 1 Introduction Multi-Document Summarization is the task of generating a summary from a cluster of related documents. State-of-the-art approaches to multi-document summarization are primarily either graph-based (Liao et al., 2018; Li et al., 2020; Pasunuru et al., 2021), leveraging graph neural networks to connect information between the documents, or hierarchical (Liu and Lapata, 2019a; Fabbri et al., 2019; Jin et al., 2020), building intermediate representations of individual documents and then aggregating information across. While effective, these models either require domain-specific additional information e.g. Abstract Meaning Representation (Liao et al., 2018), or discourse graphs (Christensen et al., 2013; Li et al., 2020), or use dataset-specific, customized architectures, making it difficult to leverage pretrained language models. Simultaneously, recent pretrained language models (typically encoder-decoder transformers)  Work mainly done during an internship at AI2. have shown the advantages of pretraining and transfer learning for generation and summarization (Raf-fel et al., 2020; Lewis et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020). Yet, existing pretrained models either use single-document pretraining objectives or use encoder-only models that do not work for generation tasks like summarization (e.g., CDLM, Caciularu et al., 2021). Therefore, we argue that these pretrained models are not necessarily the best fit for multi-document summarization. Alternatively, we propose a simple pretraining approach for multi-document summarization, reducing the need for dataset-specific architectures and large fine-tuning labeled data (See Figure 1 to compare with other pretrained mod-els). Our method is designed to teach the model to identify and aggregate salient information across a cluster of related documents during pretraining. Specifically, our approach uses the Gap Sentence Generation objective (GSG) (Zhang et al., 2020), i.e. masking out several sentences from the input document, and recovering them in order in the decoder. We propose a novel strategy for GSG sentence masking which we call, Entity Pyramid, inspired by the Pyramid Evaluation method (Nenkova and Passonneau, 2004). With Entity Pyramid, we mask salient sentences in the entire cluster then train the model to generate them, encouraging it to find important information across documents and aggregate it in one summary. ity, PRIMERA achieves superior performance compared with prior state-of-the-art pretrained models, as well as dataset-specific models in both few-shot and full fine-tuning settings. PRIMERA performs particularly strong in zeroand few-shot settings, significantly outperforming prior state-of-the-art up to 5 Rouge-1 points with as few as 10 examples. Our contributions are summarized below: 1. We release PRIMERA , the first pretrained generation model for multi-document inputs with focus on summarization. 2. We propose Entity Pyramid, a novel pretraining strategy that trains the model to select and aggregate salient information from documents. 3. We extensively evaluate PRIMERA on 6 datasets from 3 different domains for zero-shot, few-shot and fully-supervised settings. We show that PRIMERA outperforms current state-of-the-art on most of these evaluations with large margins. In this section, we discuss our proposed model PRIMERA , a new pretrained general model for multi-document summarization. Unlike prior work, PRIMERA minimizes dataset-specific modeling by simply concatenating a set of documents and processing them with a general efficient encoder-decoder transformer model (2.1). The underlying transformer model is pretrained on an unlabeled multi-document dataset, with a new entity-based sentence masking objective to capture the salient information within a set of related documents (2.2). Our goal is to minimize dataset-specific modeling to leverage general pretrained transformer models for the multi-document task and make it easy to use in practice. Therefore, to summarize a set of related documents, we simply concatenate all the documents in a single long sequence, and process them with an encoder-decoder transformer model. Since the concatenated sequence is long, instead of more standard encoder-decoder transformers like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), we use the Longformer-Encoder-Decoder (LED) Model (Beltagy et al., 2020), an efficient transformer model with linear complexity with respect to the input length. 2 LED uses a sparse local+global attention mechanism in the encoder self-attention side while using the full attention on decoder and cross-attention. When concatenating, we add special document separator tokens ( <doc-sep> ) between the documents to make the model aware of the document boundaries (Figure 2). We also assign global attention to these tokens which the model can use to share information across documents (Caciularu et al., 2021) (see 5 for ablations of the effectiveness of this input structure and global attention). In summarization, task-inspired pretraining objectives have been shown to provide gains over general-purpose pretrained transformers (PEGASUS ; Zhang et al., 2020). In particular, PEGASUS introduces Gap Sentence Generation (GSG) as a pretraining objective where some sentences are masked in the input and the model is tasked to generate them. Following PEGASUS , we use the GSG objective, but introduce a new masking strategy designed for multi-document summarization. As in GSG, we select and mask out m summary-like sentences from the input documents we want to summarize, i.e. every selected sentence is replaced by a 2 We use LED and not other efficient transformers like Bigbird-P EGASUS (Zaheer et al., 2020) for two reasons, the first is that BigBird's global attention can't be assigned to individual tokens in the middle of the sequence, which is important for the representation of long documents as shown in Caciularu et al. (2021). Second, because pretrained checkpoints are available for LED, while BigBird-P EGASUS released the already fine-tuned checkpoints. Document #1 Wildfires have burned across tens of thousands of acres of parchedterraininColorado, spurringthousandsofevacuations...(0.107)..., residents have sought shelter in middle schools, and local officials fear tourists usually drawn to the region for the summer may not come. Document #2 ... In Colorado's southwest, authorities have shuttered the San Juan National Forest in southwestern Colorado and residents of more than 2,000 homes were forced to evacuate.(0.187) No homes had been destroyed ... Under current conditions, one abandoned campfire or spark could cause a catastrophicwildfire,...,withhumanlifeandproperty,saidSanJuanNationalForestFireStaffOfficerRichardBustamante ... Document #3 The Buffalo Fire west of Denver is ... Several wildfires in Colorado have prompted thousands of home evacuations ...(0.172)... Nearly 1,400 homes have been evacuated in Summit County, Colorado, ...(0.179)... Under currentconditions,oneabandonedcampfireorsparkcouldcauseacatastrophicwildfire,...,withhumanlifeandproperty,saidRichardBustamante,SJNFforestfirestaffofficer... Entities with High Frequency Colorado, 416, Tuesday, Wildfires, San Juan National Forest,... single token [sent-mask] in the input, and train the model to generate the concatenation of those sentences as a pseudo-summary (Figure 2). This is close to abstractive summarization because the model needs to reconstruct the masked sentences using the information in the rest of the documents. The key idea is how to select sentences that best summarize or represent a set of related input documents (which we also call a cluster), not just a single document as in standard GSG. Zhang et al. (2020) use three strategies Random, Lead (first m sentences), and Principle. The Principle method computes sentence salience score based on ROUGE score of each sentence, s i , w.r.t the rest of the document ( D/ { s i } ), i.e. Score( s i ) = ROUGE ( s i , D/ { s i } ) . Intuitively, this assigns a high score to the sentences that have a high overlap with the other sentences. However, we argue that a naive extension of such strategy to multi-document summarization would be sub-optimal since multi-document inputs typically include redundant information, and such strategy would prefer an exact match between sentences, resulting in a selection of less representative information. For instance, Figure 3 shows an example of sentences picked by the Principle strategy (Zhang et al., 2020) vs our Entity Pyramid approach. The figure shows a cluster containing three news articles discussing a wildfire happened in Corolado, and the pseudo-summary of this cluster should be related to the location, time and consequence of the wildfire, but with the Principle strategy, the non-salient sentences quoting the words from an officer are assigned the highest score, as the exact same sentence appeared in two out of the three articles. In comparison, instead of the quoted words, our strategy selects the most representative sentences in the cluster with high frequency entities. To address this limitation, we propose a new masking strategy inspired by the Pyramid Evaluation framework (Nenkova and Passonneau, 2004) which was originally developed for evaluating summaries with multiple human written references. Our strategy aims to select sentences that best represent the entire cluster of input documents. Pyramid Evaluation The Pyramid Evaluation method (Nenkova and Passonneau, 2004) is based on the intuition that relevance of a unit of information can be determined by the number of references (i.e. gold standard) summaries that include it. The unit of information is called Summary Content Unit (SCU); words or phrases that represent single facts. These SCUs are first identified by human annotators in each reference summary, and they receive a score proportional to the number of reference summaries that contain them. A Pyramid Score for a candidate summary is then the normalized mean of the scores of the SCUs that it contains. One advantage of the Pyramid method is that it directly assesses the content quality. Entity Pyramid Masking Inspired by how content saliency is measured in the Pyramid Evaluation, we hypothesize that a similar idea could be applied in multi-document summarization to identify salient sentences for masking. Specifically, for a cluster with multiple related documents, the more documents an SCU appears in, the more salient that information should be to the cluster. Therefore, it should be considered for inclusion in the pseudo-summary in our masked sentence generation objective. However, SCUs in the original Pyramid Evaluation are human-annotated, which is not feasible for large scale pretraining. As a proxy, we explore leveraging information expressed as named entities, since they are key building blocks in extracting information from text about events/objects and the relationships between their participants/parts (Ju-rafsky and Martin, 2009). Following the Pyramid 5247 0.200 Doc 1 Sent 2 Entity 1 Entity 2 Entity 3 Entity 4 Entity 5 Entity 6 Entity 7 Entity 8 frequency: 1 frequency: 2 frequency: 3 frequency: 4 Doc 1: {Entity 1, Entity 2, Entity 6} Doc 2: {Entity 1, Entity 2, Entity 3, Entity 5, Entity 7} Doc 3: {Entity 1, Entity 2, Entity 3, Entity 4, Entity 5} Doc 4: {Entity 1, Entity 3, Entity 4, Entity 8} Entity 1 Entity 4 Doc 1 Sent 1 Doc 2 Sent 10 ... framework, we use the entity frequency in the cluster as a proxy for saliency. Concretely, as shown in Fig. 4, we have the following three steps to select salient sentences in our masking strategy: 1. Entity Extraction . We extract named entities using SpaCy (Honnibal et al., 2020). 3 2. Entity Pyramid Estimation . We then build an Entity Pyramid for estimating the salience of entities based on their document frequency, i.e. the number of documents each entity appears in. Next, we iteratively select entities from top of the pyramid to bottom (i.e., 3 Note that entity information is only used at pretraining time. 3. Sentence Selection . Similar to the Pyramid evaluation framework, we identify salient sentences with respect to the cluster of related documents. Algorithm 1 shows the sentence selection procedure. As we aim to select the entities better representing the whole cluster instead of a single document, we first remove all entities from the Pyramid that appear only in one document. This is unlike some prior work (e.g. Pasunuru et al. (2021)) that utilize additional information (like named entities, coref, discourse, or AMR) at fine-tuning and inference time. highest to lowest frequency), and then select sentences in the document that include the entity as the initial candidate set. Finally, within this candidate set, we find the most representative sentences to the cluster by measuring the content overlap of the sentence w.r.t documents other than the one it appears in. This final step supports the goal of our pretraining objective, namely to reconstruct sentences that can be recovered using information from other documents in the cluster, which encourages the model to better connect and aggregate information across multiple documents. Following Zhang et al. (2020) we use ROUGE scores (Lin, 2004) as a proxy for content overlap. For each sentence s i , we specifically define a Cluster ROUGE score as Score ( s i ) = (cid:80) { doc j  C,s i (cid:54) doc j } ROUGE ( s i , doc j ) Where C is the cluster of related documents. Note that different from the importance heuristic defined in PEGASUS (Zhang et al., 2020), Entity Pyramid strategy favors sentences that are representative of more documents in the cluster than the exact matching between fewer documents (See Figure 3 for a qualitative example.) . The benefit of our strategy is shown in an ablation study (5). We aim to answer the following questions:  Q1: How does PRIMERA perform, compared with existing pre-trained generation models in zero-and few-shot settings? See 4.2. Q2: How does PRIMERA perform, compared with current state-of-the-art models, in the fully supervised setting? See 4.5. Q3: How much is the contribution of each component in PRIMERA , i.e. input structure, pretraining, and masking strategy? See 5. Q4: What is the effect of our entity pyramid 5248 Dataset #Examples #Doc/C Len src Len summ Newshead (2020) 360K 3.5 1734 -Multi-News (2019) 56K 2.8 1793 217 Multi-Xscience (2020) 40K 4.4 700 105 Wikisum* (2018) 1.5M 40 2238 113 WCEP-10 (2020) 10K 9.1 3866 28 DUC2004 (2005) 50 10 5882 115 arXiv (2018) 214K 5.5 6021 272 Table 1: The statistics of all the datasets we explore in this paper. PEGASUS? See 5. Q5: Is PRIMERA able to capture salient information and generate fluent summaries? See 6. With these goals, we explore the effectiveness of PRIMERA quantitatively on multi-document summarization benchmarks, verify the improvements by comparing PRIMERA with multiple existing pretrained models and SOTA models, and further validate the contribution of each component with carefully controlled ablations. An additional human evaluation is conducted to show PRIMERA is able to capture salient information and generate more fluent summaries. Implementation Details We use the Longformer-Encoder-Decoder (LED) (Belt-agy et al., 2020) large as our model initialization, The length limits of input and output are 4096 and 1024, respectively, with sliding window size as w = 512 for local attention in the input. (More implementation details of pretraining process can be found in Appx A) Pretraining corpus For pretraining, our goal is to use a large resource where each instance is a set of related documents without any ground-truth summaries. The Newshead dataset (Gu et al., 2020) (row 1, Table 1) is an ideal choice; it is a relatively large dataset, where every news event is associated with multiple news articles. Evaluation Datasets We evaluate our approach on wide variety of multi-document summarization datasets plus one single document dataset from various domains (News, Wikipedia, and Scientific literature). See Table 1 for dataset statistics and Appx. B for details of each dataset. Evaluation metrics Following previous works (Zhang et al., 2020), we use ROUGE scores (R-1, -2, and -L), which are the standard evaluation metrics, to evaluate the downstream task of multi-document summarization. 4 For better readability, we use AVG ROUGE scores (R-1, -2, and -L) for evaluation in the few-shot setting. Many existing works in adapting pretrained models for summarization require large amounts of fine-tuning data, which is often impractical for new domains. In contrast, since our pretraining strategy is mainly designed for multi-document summarization, we expect that our approach can quickly adapt to new datasets without the need for significant fine-tuning data. To test this hypothesis, we first provide evaluation results in zero and few-shot settings where the model is provided with no, or only a few (10 and 100) training examples. Obtaining such a small number of examples should be viable in practice for new datasets. Comparison To better show the utility of our pretrained models, we compare with three state-of-the-art pretrained generation models: BART (Lewis et al., 2020) 5 , PEGASUS (Zhang et al., 2020) and Longformer-Encoder-Decoder(LED) (Beltagy et al., 2020). These pretrained models have been shown to outperform dataset-specific models in summarization (Lewis et al., 2020; Zhang et al., 2020), and because of pretraining, they are expected to also work well in the few-shot settings. As there is no prior work doing few-shot and zero-shot evaluations on all the datasets we consider, and also the results in the few-shot setting might be influenced by sampling variability (especially with only 10 examples) (Bragg et al., 2021), we run the same experiments for the compared models five times with different random seeds (shared with all the models), with the publicly available checkpoints . 6 Similar to Pasunuru et al. (2021), the inputs of all the models are the concatenations of the documents within the clusters (in the same order), each document is truncated based on the input length limit divided by the total number of documents so 4 We use https://github.com/google-research/googleresearch/tree/master/rouge with default stemmer settings. 5 Pilot experiments comparing BART and T5 showed BART to outperform T5 on the few-shot evaluation of Multi-News (with AVG ROUGE of 23.5/26.4 (T5) v.s. 25.2/26.7 (BART) for 10/100 training examples, respectively). Thus, we are using BART as one of the baselines. 6 Checkpoints from https://huggingface.co/models 5249 Models Multi-News(256) Multi-XSci(128) WCEP(50) WikiSum(128) arXiv(300) DUC2004 (128) R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L PEGASUS (cid:63) (Zhang et al., 2020) 36.5 10.5 18.7 ----28.1 6.6 17.7 -PEGASUS (our run) 32.0 10.1 16.7 27.6 4.6 15.3 33.2 12.7 23.8 24.6 5.5 15.0 29.5 7.9 17.1 32.7 7.4 17.6 BART (our run) 27.3 6.2 15.1 18.9 2.6 12.3 20.2 5.7 15.3 21.6 5.5 15.0 29.2 7.5 16.9 24.1 4.0 15.3 LED (our run) 17.3 3.7 10.4 14.6 1.9 9.9 18.8 5.4 14.7 10.5 2.4 8.6 15.0 3.1 10.8 16.6 3.0 12.0 PRIMERA (our model) 42.0 13.6 20.8 29.1 4.6 15.7 28.0 10.3 20.9 28.0 8.0 18.0 34.6 9.4 18.3 35.1 7.2 17.9 Table 2: Zero-shot results. that all documents are represented in the input. 7 To preserve the same format as the corresponding pretrained models, we set the length limit of output for BART and PEGASUS exactly as their pretrained settings on all of the datasets (except for the zero-shot experiments, the details can be found in Sec.4.3). Regarding length limit of inputs, we tune the baselines by experimenting with 512, 1024, 4096 on Multi-News dataset in few-shot setting (10 data examples), and the model with length limit 512(P EGASUS )/1024(BART) achieves the best performance, thus we use this setting (de-tailed experiment results for different input lengths can be found in Appx. C.1). We use the same length limit as our model for the LED model, i.e. 4096/1024 for input and output respectively, for all the datasets. For zero-shot 8 abstractive summarization experiments, since the models have not been trained on the downstream datasets, the lengths of generated summaries mostly depend on the pretrained settings. Thus to better control the length of generated summaries and for a fair comparison between all models, following Zhu et al. (2021), we set the 7 Pilot experiments show simple truncation results in inferior performance, which is in line with Pasunuru et al. (2021). length limit of the output at inference time to the average length of gold summaries. 9 Exploring other approaches to controlling length at inference time (e.g., Wu et al., 2021) is an orthogonal direction, which we leave for future work. Table 2 shows the performance comparison among all the models. Results indicate that our model achieves substantial improvements compared with all the three baselines on most of the datasets. As our model is pretrained on clusters of documents with longer input and output, the benefit is stronger on the dataset with longer summaries, e.g. Multi-News and arXiv. Comparing PEGASUS and BART models, as the objective of PEGASUS is designed mainly for summarization tasks, not surprisingly it has relatively better performances across different datasets. Interestingly, LED un-derperforms other models, plausibly since part of the positional embeddings (1k to 4k) are not pretrained. Encouragingly, our model performs the best, demonstrating the benefits of our pretraining strategy for multi-document summarization. Compared with the strict zero-shot scenario, few-shot experiments are closer to the practical scenarios, as it is arguably affordable to label dozens of examples for almost any application. We fine-tune all of the four models on different subsets with 10 and 100 examples, and the results are shown in Figure 5. (hyperparameter settings in Appx. D.1) Since R-1, -2, and -L show the same trend, we only present the average of the three metrics in the figure for brevity (full ROUGE scores can be found in Appx. Table 8) To show the generality, all the results of few-shot experiments are the average over 5 runs on different subsets (shared by all the models). The result of each run is obtained by the best' model chosen based on the ROUGE scores on a randomly sampled few-shot validation set with the same number of examples as the training set, which is similar with Zhang et al. (2020). Note that their reported best models have been selected based on the whole validation set which may give PEGASUS some advantage. Nevertheless, we argue that sampling few-shot validation sets as we do here is closer to real few-shot scenarios (Bragg et al., 2021). Our model outperforms all baselines on all of the datasets with 10 and 100 examples demonstrating the benefits of our pretraining strategy and input structure. Comparing the performances of our model with the different number of training data fed in, our model converges faster than other models with as few as 10 data examples. To show the advantage of our pretrained model when there is abundant training data, we also train the model with the full training set (hyperparame-ter settings can be found in Appx. D.2). Table 3 shows the performance comparison with previous state-of-the-art 10 , along with the results of previous SOTA. We observe that PRIMERA achieves state-of-the-art results on Multi-News, WCEP, and arXiv, while slightly underperforming the prior work on Multi-XScience (R-1). One possible explanation is that in Multi-XScience clusters have less overlapping information than in the corpus on which PRIMERA was pretrained. In particular, the source documents in this dataset are the abstracts of all the publications cited in the related work paragraphs, which might be less similar to each other and the target related work(i.e., their summary) . PRIMERA 9 We re-evaluate the generated summaries of the models from Lu et al. (2020) for Multi-XScience, as we use a different version of ROUGE. outperforms the LED model (State-of-the-art) on the arXiv dataset while using a sequence length 4x shorter (4K in PRIMERA v.s. 16K in LED), further showing that the pretraining and input structure of our model not only works for multi-document summarization, but can be also effective for summarizing single documents having multiple sections. We conduct ablation studies on the Multi-News dataset in few-shot setting, to validate the contribution of each component in our pretrained models. Input structure: In Figure 6 (a) we observe the effectiveness of both pretraining and the input structure ( <doc-sep> tokens between documents and global attention on them). Sentence masking strategy: To isolate the effect of our proposed pretraining approach, we compare with a model with exactly the same architecture when pretrained on the same amount of data but using the PEGASUS (Zhang et al., 2020) masking strategy instead of ours. In other words, we keep all the other settings the same (e.g., data, length limit of input and output, pretraining dataset, input structure, as well as the separators) and only modify the pretraining masking strategy. We run the same experiments under zero-/few-shot scenar-5251 ios on the Multi-News dataset as in 4.2, and the results are shown in Figure 6 (b). The model pretrained with our Entity Pyramid strategy shows a clear improvement under few-shot scenarios. We also conduct human evaluations to validate the effectiveness of PRIMERA on DUC2007 and TAC2008 (Dang and Owczarzak, 2008) datasets in the few-shot setting (10/10/20 examples for train/valid/test). Both datasets consist of clusters of news articles, and DUC2007 contains longer inputs (25 v.s. 10 documents/cluster) and summaries (250 v.s. 100 words). Since the goal of our method is to enable the model to better aggregate information across documents, we evaluate the content quality of the generated summaries following the original Pyramid human evaluation framework (Nenkova and Passonneau, 2004). In addition, we also evaluate the fluency of generated summaries following the DUC guidelines. 12 Settings Three annotators 13 are hired to do both Pyramid Evaluation and Fluency evaluation, they harmonize the standards on one of the examples. Specifically, for each data example, we provide three anonymized system generated summaries, along with a list of SCUs. The annotators are asked to find all the covered SCUs for each summary, and score the fluency in terms of Grammaticality, Referential clarity and Structure & Coherence, according to DUC human evaluation guidelines, with a scale 1-5 (worst to best). They are also suggested to make comparison between three generated summaries into consideration when scoring the fluency. To control for the ordering effect of the given summaries, we re-order the three summaries for each data example, and ensure the chance of their appearance in different order is the same (e.g. BART appears as summary A for 7 times, B for 7 times and C for 6 times for both datasets). The instruction for human annotation can be found in Figure 7 and Figure 8 in the appendix. Annotators were aware that annotations will be used solely for computing aggregate human evaluation metrics and reporting in the scientific paper. cause PEGASUS is a task-specific model for abstractive summarization, and LED has the same architecture and length limits as our model with the parameters inherited from BART, which is more comparable with our model than vanilla BART. Pyramid Evaluation Both TAC and DUC datasets include SCU (Summary Content Unit) annotations and weights identified by experienced annotators. We then ask 3 annotators to make a binary decision whether each SCU is covered in a candidate summary. Following Nenkova and Pas-sonneau (2004), the raw score of each summary is then computed by the sum of weights of the covered SCUs, i.e. S r = (cid:80) SCU w i I ( SCU i ) , where I ( SCU i ) is an indicator function on whether SCU i is covered by the current summary, and w i is the weight of SCU i . In the original pyramid evaluation, the final score is computed by the ratio of S r to the maximum possible weights with the same number of SCUs as in the generated summaries. However, the total number of SCUs of generated summaries is not available in the simplified annotations in our design. To take consideration of the length of generated summaries and make a fair comparison, instead, we compute Recall, Precision and F-1 score regarding lengths of both gold references and system generated summaries as R= S r len ( gold ) ; P= S r len ( sys ) ; F1= 2  R  P ( R + P ) Fluency Evaluation Fluency results can be found in Table 5, and PRIMERA has the best performance on both datasets in terms of all aspects. Only 5252 for Grammaticality PRIMERA's top performance is matched by PEGASUS. Neural Multi-Document Summarization These models can be categorized into two classes, graph-based models (Yasunaga et al., 2017; Liao et al., 2018; Li et al., 2020; Pasunuru et al., 2021) and hierarchical models (Liu and Lapata, 2019a; Fabbri et al., 2019; Jin et al., 2020). Graph-based models often require auxiliary information (e.g., AMR, discourse structure) to build an input graph, making them reliant on auxiliary models and less general. Hierarchical models are another class of models for multi-document summarization, examples of which include multi-head pooling and inter-paragraph attention (Liu and Lapata, 2019a), MMR-based attention (Fabbri et al., 2019; Mao et al., 2020), and attention across representations of different granularity (words, sentences, and documents) (Jin et al., 2020). Prior work has also shown the advantages of customized optimization in multi-document summarization (e.g., RL; Su et al., 2021). Such models are often dataset-specific and difficult to develop and adapt to other datasets or tasks. Pretrained Models for Summarization Pretrained language models have been successfully applied to summarization, e.g., BERTSUM (Liu and Lapata, 2019b), BART (Lewis et al., 2020), T5 (Raffel et al., 2020). Instead of regular language modeling objectives, PEGASUS (Zhang et al., 2020) introduced a pretraining objective with a focus on summarization, using Gap Sentence Generation, where the model is tasked to generate summary-worthy sentences, and Zou et al. (2020) proposed different pretraining objectives to reinstate the original document, specifically for summarization task as well. Contemporaneous work by Rothe et al. (2021) argued that task-specific pretraining does not always help for summarization, however, their experiments are limited to single-document summarization datasets. Pretraining on the titles of HTMLs has been recently shown to be useful for few-shot short-length single-document summarization as well (Aghajanyan et al., 2021). Goodwin et al. (2020) evaluate three state-of-the-art models (BART, PEGASUS , T5) on several multi-document summarization datasets with low-resource settings, showing that abstractive multi-document summarization remains challenging. Efficient pretrained transformers (e.g., Longformer (Beltagy et al., 2020) and BigBird (Zaheer et al., 2020) that can process long sequences have been also proven successful in summarization, typically by the ability to process long inputs, connecting information across the entire sequence. CDLM (Caciularu et al., 2021) is a follow-up work for pretraining the Longformer model in a cross-document setting using global attention on masked tokens during pretraining. However, this model only addresses encoder-specific tasks and it is not suitable for generation. In this work, we show how efficient transformers can be pretrained using a task-inspired pretraining objective for multi-document summarization. Our proposed method is also related to the PMI-based token masking Levine et al. (2020) which improves over random token masking outside summarization. In this paper, we present PRIMERA a pre-trained model for multi-document summarization. Unlike prior work, PRIMERA minimizes dataset-specific modeling by using a Longformer model pretrained with a novel entity-based sentence masking objective. The pretraining objective is designed to help the model connect and aggregate information across input documents. PRIMERA outperforms prior state-of-the-art pre-trained and dataset-specific models on 6 summarization datasets from 3 different domains, on zero, few-shot, and full fine-tuning setting. PRIMERA 's top performance is also revealed by human evaluation. In zero-shot setting, we can only control the output length of generated summaries at inference time by specifying a length limit during decoding. Exploring a controllable generator in which the desired length can be injected as part of the input is a natural future direction. Besides the summarization task, we would like to explore using PRIMERA for other generation tasks with multiple documents as input, like multi-hop question answering. While there is limited risk associated with our work, similar to existing state-of-the-art generation models, there is no guarantee that our model will always generate factual content. Therefore, caution must be exercised when the model is deployed in practical settings. Factuality is an open problem in existing generation models.
method objective abstain abstain abstain other abstain abstain abstain abstain abstain objective method abstain abstain result abstain abstain objective objective method result abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method method abstain abstain abstain abstain result abstain result other abstain abstain other abstain result result abstain abstain result abstain abstain abstain result other abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain other abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other abstain other other other other other other other other other other other other other objective abstain method abstain abstain abstain abstain method abstain objective method abstain abstain