Neural-based summarization models suffer from the length limitation of text encoder. Long documents have to been truncated before they are sent to the model, which results in huge loss of summary-relevant contents. To address this issue, we propose the sliding selector network with dynamic memory for extractive summarization of long-form documents, which employs a sliding window to extract summary sentences segment by segment. Moreover, we adopt memory mechanism to preserve and update the history information dynamically, allowing the semantic flow across different windows. Experimental results on two large-scale datasets that consist of scientific papers demonstrate that our model substantially outperforms previous state-of-the-art models. Besides, we perform qualitative and quantitative investigations on how our model works and where the performance gain comes from. 1 1 Introduction Text summarization is an important task of natural language processing which aims to distil salient contents from a textual document. Existing summarization models can be roughly classified into two categories, which are abstractive and extractive. Abstractive summarization usually adopts natural language generation technology to produce a word-by-word summary. In general, these approaches are flexible but may yield disfluent summaries (Liu and Lapata, 2019a). By comparison, extractive approaches aim to select a subset of the sentences in the source document, thereby enjoying better fluency and efficiency (Cao et al., 2017). Paragraph 1 : Medical tourism is illustrated as occurrence in which individuals travel abroad to receive healthcare services. It is a multibillion dollar industry and countries like India, Thailand, Israel, Singapore,  Paragraph 2 : The prime driving factors in medical tourism are increased medical costs, increased insurance premiums, increasing number of uninsured or partially insured individuals in developed countries,   Paragraph 5 : It is generally presumed in marketing that products with similar characteristics will be equally preferred by the consumers, however, attributes, which make the product similar to other products, will not. to achieve desired performance when directly applied in long-form documents, such as scientific papers. This inferior performance is partly due to the truncation operation, which inevitably leads to information loss, especially for extractive models because parts of gold sentences would be inaccessible. In addition, the accurate modeling of long texts remains a challenge (Frermann and Klemen-tiev, 2019). A practical solution for this problem is to use a sliding window to process documents separately. This approach is used in other NLP tasks, such as machine reading comprehension (Wang et al., 2019b). However, such a paradigm is not suitable for summarization task because the concatenation of summaries that are independently extracted from local contexts is usually inconsistent with the gold summary of the entire document. Figure 1 shows an example to illustrate this problem. The core topic of the source document is  medical tourism , which is discussed in Paragraphs 1 and 2. However, the 5-th paragraph is mainly about  consumer and product . As a consequence, the paragraph-by-paragraph extraction approach might produce a both repetitive and noisy summary. Under this circumstance, the supervised signals will have a negative effect on model behaviors because understanding why Paragraph 5 should output an empty result without information conveying from previous texts is confused for the model. In this paper, we propose a novel extractive summarization model for long-form documents. We split the input document into multiple windows and encode them with a sliding encoder sequentially. During this process, we introduce a memory to preserve salient information learned from previous windows, which is used to complete and enrich local texts. Intuitively, our model has the following advantages: 1) In each window, the text encoder processes a relatively short segment, thereby yielding more accurate representations. 2) The local text representations can capture beyond-window contextual information via the memory module. 3) The previous selection results are also parameterized in the memory block, allowing the collaboration among summary sentences. To sum up, our contributions are threefold. (1) We propose a novel extractive summarization model that can summarize documents of arbitrary length without truncation loss. Also, it employs the memory mechanism to address context fragmentation. To the best of our knowledge, we are the first to propose applying memory networks into extractive text summarization task. (2) The proposed framework (i.e., a sliding encoder combined with dynamic memory) provides a general solution for summarizing long documents and can be easily extended to other abstractive and extractive summarization models. (3) Our model achieves the state-of-the-art results on two widely used datasets for long document summarization. Moreover, we conduct extensive analysis to understand how our model works and where the performance gain comes from. Neural Extractive Summarization . Neural-networks have become the dominant approach for extractive summarization. Existing studies usually formulate this task as sentence labelling (Dong et al., 2018; Nallapati et al., 2016; Zhang et al., 2019) or sentence ranking (Narayan et al., 2018). Among them, recurrent neural networks (Cheng and Lapata, 2016; Zhou et al., 2018), Transformer encoder (Wang et al., 2019a), or graph neural networks (Wang and Liu, 2020, Xu et al., 2020, Cui et al., 2020) (Wang et al., 2020; Xu et al., 2019; Cui et al., 2020) have been used to learn sentence representation. Recently, pre-trained language model (e.g. BERT (Devlin et al., 2018)) has provided substantial performance gain for extractive summarization. Liu and Lapata (2019b) modified standard BERT for document modelling. Xu et al. (2019) used a span-BERT to perform span-level summarization. Zhong et al. (2020) regarded document summarization as a semantic matching task and used a Siamese-BERT as the matching model. However, the valid length of standard BERT is only 512, which means most of them can hardly generalize to long-form documents effectively. Long Document Summarization . Recent years have seen a surge of interest on long document summarization, especially scientific publications. Celikyilmaz et al. (2018) used a multi-agent framework to boost the encoder performance. Cohan et al. (2018) proposed a hierarchical network that incorporates the discourse structures into the encoder and decoder. Xiao and Carenini (2019) proposed to model the local and global contexts jointly. Cui et al. (2020) proposed a hybrid model that employs a neural topic model (NTM) to infer latent topics as a kind of global information. Despite their success, these approaches still face the input length limitation and the difficulty in encoding long texts accurately. In comparison, our model addresses these problems with a novel segment-wise extraction way and can summarize arbitrarily long documents without any content truncation. Memory Networks . Memory network (Weston et al., 2015) is a general framework that employs a memory bank to model long-term information. Due to its flexible architecture and superior adaptability, it has been applied into various NLP scenarios, such as text classification (Zeng et al., 2018), question answering (Kumar et al., 2016; Xiong et al., 2016), and sentiment analysis (Tang et al., 2016). In this study, we leverage a memory module capture beyond-window when performing segment-level summarization. To the best of our knowledge, memory networks have never been applied into extractive summarization task. This section describes our model, namely, the Sliding Selector Network with Dynamic Memory (SSN-DM), of which Figure 2 gives an overall architecture. Formally, given a document D of arbitrary length, we first split D into multiple segments according to the pre-defined window length. Then, we use a BERT encoder to sequentially encode each segment and select salient sentences. During this process, a memory module is applied to achieve the information flow across different windows. Finally, the extracted sentences are aggregated to generate the final summary. We elucidate each module in the following subsections. Let seg k = s k 1 , s k 2 , . . . , s kn be the kth window consisting of n sentences. We encode the window text with a pre-trained BERT, which has been proven effective on extractive summarization task (Liu and Lapata, 2019b; Xu et al., 2019; Cui et al., 2020). Following previous studies, we modify the standard BERT by inserting [ CLS ] and [ SEP ] tokens into the beginning and end of each sentence, respectively. where w ki,j denotes the jth word of the ith sentence. OB = { h k 1 ,CLS , h k 1 , 2 , . . . , h kn,SEP } denotes the representations of each token learned by BERT. We regard the hidden states of [ CLS ] tokens H k = { h k 1 ,CLS , h k 2 ,CLS , . . . , h kn,CLS } as the corresponding sentence representations. On top of BERT encoder, we add an additional layer to incorporate two types of structural information. The first part is the position information of the current window. In our segment-wise encoding, the position embeddings equipped in BERT are recalculated in each window, thereby losing the exact position of each token in the entire document. This positional bias may lead to inferior performance (Zhong et al., 2019; Dai et al., 2019). To address this problem, we assign a window-level position encoding to each window as a complementary feature, indicating its relative position in the document. In addition, we further introduce a group of section (e.g., introduction, conclusion) embeddings to capture the discourse information, which has been proved an important feature for scientific papers summarization (Cohan et al., 2018). Combining where e kw indicates the kth window-level position embedding, and e s the section embedding. Both of them are randomly initialized and learned as a part of the model. Throughout the paper, W  represents trainable parameter matrix. Noticeably, the section features might not be generally available for long texts of other genres. Therefore, in our experiments, we consider e s as an optional setting and conduct quantitative investigations on Section 5 to probe into its effect on model performance. After encoding the window text, we infuse the history information of previous texts into the learned representations H k via a memory module. Let M k  R l  d m be the memory block in the kth window that preserves salient information of previous k  1 windows, where l represents the number of memory slots and d m represents the dimension of memory vector. M 0 is initialized with fixed values in the first window and then updated in the learning process dynamically. The detail of this part is explained in Section 3.4. We use a graph neural network to model the interaction between memory module and the current window. Concretely, we first construct a bipartite graph that consists of l memory nodes and n sentence nodes, whose embeddings are initialized with M k and H k , respectively. Then, we use graph attention network (GAT; Velickovic et al., 2018) to encode this graph. Given a sentence node h i , we update its representation by aggregating its neighboring nodes, as shown as follows, z ki,j = LeaklyRelu ( W a [ h ki ; SG ( m kj )]) ,  i,j = exp ( z ki,j ) (cid:80) lj =1 exp ( z ki,j ) ,  h ki = (cid:107) Tt =1 l (cid:88) j =1 tanh (  ti,j W tc SG ( m kj )) , (3) where  i,j denotes the attention weight from node h ki to node m kj . Multi-head attention is applied to stabilize the calculation process. Function SG (  ) stands for stop-gradient operation. We refer  H k and  M k to the sentence representations and memory vectors after graph propagation, respectively. During the graph interaction, the sentence representations are completed and enriched by history information and vice versa. Empirical observations of prior research (Tang et al., 2016; Zeng et al., 2018) have shown that stacking multiple memory layers can bring further performance gain. Similarly, in our model, the multi-hops setting can be achieved by increasing the graph iteration number, i.e., repeating the GAT calculation process (Eq. 3). We have obtained the sentence representations H k derived from window text, and its extended version  H k enriched by memory information. Given ith sentence, we send h ki and  h ki into a MLP classifier to compute its summary label. where  y i represents the predicted probability of ith sentence, and  represents the point-wise operation. f o is a feed-forward network with three hidden layers. We construct interaction features between  h ki and h ki to capture the importance of ith sentence in both current segment and history context. The training objective of the model is to minimize the binary cross-entropy loss given the predictions and ground truth sentence labels, i.e., L =  (cid:80) y i log ( y i ) + (1  y i ) log (1   y i ) After processing the entire document, we rank all the sentences and select top-k as the final summary, where k is a hyperparameter set according to the average length of reference summaries. It worth noting that the memory module also acts as an intermediary to make the sentence scores of different windows comparable. Now we explain the learning process of memory module. Figure 3 presents the information flow of our model. In each window, after the prediction layer, we update the memory values with two inputs. First, recall that in GAT calculation, the updated memory vectors  M k has also encoded the contextual information of the current window during the interaction with H k . Therefore, we combine  M k and M k with gating mechanism (Chung et al., 2014). ki = tanh ( W m   m ki ) , u ki =  ki  m ki + (1   ki )   m ki (5) where u ki is the liner interpolation between history memory m ki and the newly computed mi k . ki  R d m is an gate vector to modulates the information flow. The second part refers to the extraction result of the current window. We first aggregate the sentence representations with their predicted probabilities (Eq.4) to parameterize the selected sentences. Here, r ksum can be considered a sentence-level coverage vector (See et al., 2017) that records what contents has been extracted from the current window. This ensures that the following selection is informed by previous decisions. Then, we use a single feedforward layer to generate new memory M k +1 = { m k +11 , . . . , m k +1 l } for next window. Our model is particularly designed for long document summarization. For this reason, we do not conduct experiments on the widely explored news datasets consisting of relatively short documents. For example, the articles in DailyMail (Hermann et al., 2015) dataset have an average of 600 words, which can be effectively processed by most existing models. Instead, following prior research on long-form documents summarization(Cohan et al., 2018; Xiao and Carenini, 2019; Cui et al., 2020; Zhong et al., 2020), we evaluate our model on the following two large-scale scientific paper datasets. For example, the maximum length of standard BERT is 512, which means that a large proportion 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 0 ~ 250250 ~ 500500 ~ 750 750 ~ 1000 1000 ~ 1500 1500 ~ 2000 2000 ~ 3000 3000 ~ 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 0 ~ 250250 ~ 500500 ~ 750 750 ~ 1000 1000 ~ 1500 1500 ~ 2000 2000 ~ 3000 3000 ~ P r opo r ti on P r opo r ti on Position of Gold Sentences Position of Gold Sentences Figure 4: Position distribution of gold sentences on two datasets. (colored in grey) of ground-truth sentences would be inaccessible for existing state-of-the-art BERT-based summarization models. state-of-the-art summarization approaches. Pointer Generator Network (PGN; See et al., 2017) extends the standard seq2seq framework with attention, coverage, and copy mechanism. Discourse-Aware (Cohan et al., 2018) is an abstractive model particularly designed for summarizing long-form document with discourse structure. It employs a hierarchical encoder and explicitly introduces the section information of scientific papers. Seq2seq-local&global (Xiao and Carenini, 2019) is also an extractive model for long document summarization that jointly encodes local and global contexts. Match-Sum (Zhong et al., 2020) is a state-of-the-art BERT-based summarization model. It performs summary-level extraction based on the matching scores between candidate summary and the source document. Topic-GraphSum (Cui et al., 2020) introduces a joint neural topic model to explore latent topics as a kind of global information to help summarize long documents. Since Cui et al. (2020) used different data preprocessing, we repeat the experiments using the model released by the authors and preprocess the data in accordance with previous studies (Cohan et al., 2018; Xiao and Carenini, 2019) to make the results comparable. For the sliding encoder, we use the bert-base-uncased version with the hidden size of 768 and fine-tune it for all experiments. The maximum length of window is set to 512, and we segment the Models arXiv PubMed R-1 R-2 R-L R-1 R-2 R-L Lead 33.66 8.94 22.19 35.63 12.28 25.17 LexRank+ 33.85 10.73 28.99 39.19 13.89 34.59 LSA+ 29.91 7.42 25.67 33.89 9.93 29.70 Oracle* 53.88 23.05 34.90 55.05 27.48 38.66 Seq2seq-attention+ 29.30 6.00 25.56 31.55 8.52 27.38 PGN+ 32.06 9.04 25.16 35.86 10.22 29.69 Disourse-aware+ 35.80 11.05 31.80 38.93 15.37 35.21 Cheng & Lapta (2016)* 42.24 15.97 27.88 43.89 18.53 30.17 SummaRuNNer* 42.81 16.52 28.23 43.89 18.78 30.36 Seq2seq-local&global* 43.62 17.36 29.14 44.85 19.70 31.43 Match-Sum 40.59 12.98 32.64 41.21 14.91 36.75 Topic-GraphSum 44.03 18.52 32.41 45.95 20.81 33.97 SSN-DM 45.03 19.03 32.58 46.73 21.00 34.10 SSN-DM + discourse 44.90 19.06 32.77 46.52 20.94 35.20 Table 2: Rouge results on two dataets. documents with sentence as the smallest unit to alleviate semantic fragility. For the memory module, we set the number of slots to 50 and the dimension of the memory vector to 768, same with the hidden size of the encoder. The iteration number of GAT is set to 2. We use Rouge (Lin, 2004) as the evaluation metric and select the hyperparameters by grid search based on the Rouge-2 performance on validation sets. Further analysis about the impacts of hyperparameters are discussed in Section 5.2. We train our model with 2 NVIDIA V100 cards with a small batch size of 16. During the training, we use Adam (Kingma and Ba, 2015) to optimize parameters with a learning rate of 5e-4. An early-stop strategy (Caruana et al., 2000) is applied when valid loss is no longer decent. The extracted sentence number is set to 7 for arXiv dataset and 6 for PubMed dataset according to their average summary length. We report the average results over 5 runs. Table 2 presents the results of different models on two datasets. The first section includes traditional approaches and the Oracle; the second and the third sections includes abstractive and extractive models, respectively; and the last section reports ours. Our 0.00% 10.00% 20.00% 30.00% 40.00% 1 2 3 4 5 6 7 8 arXiv PubMed Window Index P r o p o r t i o n Figure 5: Proportion of sentences selected by each window. model with discourse represents that we leverage section information as additional feature (Eq. 2). Several observations deserve to be mentioned. Encoding long texts for abstractive summarization is a challenge . The vanilla seq2seq with attention model and the pointer network perform rather poorly on the two datasets. A possible reason is that most encoders experience difficulties in modeling long-range contextual dependency when encoding long texts (Vaswani et al., 2017; Frermann and Klementiev, 2019), thereby leading to the inferior performance during the generation (decoding) process. Global Information Modeling is important for summarizing long documents . We also observe that Seq2seq-local&global and Topic-Number of Slots R1 35 40 45 50 arXiv PubMed 20 40 60 80 100 150 200 35 40 45 50 128 192 256 320 384 448 512 arXiv PubMed Window Length R1 Figure 6: Impact of window length (left) and slot number (right) on model performance (R-1). GraphSum show promising results on the two datasets. Both of them explicitly model the global information (e.g., latent topics). Such observation provides a useful instruction for designing the summarization model for long documents. Our framework is effective . Our two models substantially outperform all the baselines on two datasets. Figure 5 shows the proportion of sentences selected by each window, where we can see that our model can extract contents from any position of an entire document. By contrast, BERT-Sum and Topic-GraphSum, two BERT-based strong baselines, can only select sentences from the first 512 or 768 words because their truncation setting. This superiority endows our model a higher upper bound when summarizing long documents. Discourse structure is automatically captured . The last section of Table 3 shows that the incorporation of discourse information brings no substantial performance gain for our model, though observations in previous studies (Cohan et al., 2018; Xiao and Carenini, 2019) have shown it an effective feature on arXiv and PubMed datasets. A possible reason is that our window-level position encoding has already learned such discourse information because it indicates the window's relative position in the document, while scientific papers are generally organized in specific and relatively fixed structure. This observation implies that the performance of our model does not rely on prior information of datasets. As a result, our model could be easily generalized to long texts of other genres. We conduct experiments to probe into the impact of several important hyperparameters on model performance, including window length, number of memory slots, and number of memory hops (i.e., iteration number of GAT). Impact of Window Length . Intuitively, a shorter window means more accurate text encoding. However, it will result in more segments, which is demanding for memory module. Therefore, it is important to find a balanced window length. Figure 6 (left) shows that the overall performance is enhanced when the window length increases from a small value (128). This is because that too short windows suffer from semantic fragility. However, when the window length is set to 368-512, the performance shows a stable trend, implying that the step number and text length are both in a suitable range. For the sake of efficiency, we set the window length to 512 in our experiments. Impact of Slots Numbers . Figure 6 (right) presents the Rouge-1 results on varying slot numbers. As can be seen, the curves on the two datasets are not monotonous and show a similar trend. In particular, within a particular range where l is relatively small, more slots produce better performance because the memory capacity is improving. However, such increasing trend will reach a saturation when slot number exceeds a threshold, which is 60 in our experiments. Impact of Iteration Numbers . Recall that in memory layer, we employ a GAT to calculate the interaction between the memory and the window texts. To select the best iteration number (hop number) t , we compare the performance of different t on the validation sets of two datasets. Table 3 shows when t goes from 0 to 2, the performance is slightly boosted. However, this increasing trend is not always monotonous, and a larger t does not bring further substantial gain. To balance the time cost and performance, we select t =2 for the two datasets. of memory module. To this end, we construct an ablated version by removing the memory module and then seek to observe the result difference. Case Study . Figure 7 provides a case study that compares the selection results of the ablated model and our full model. In 4-th window, the ablated model selects a repetitive sentence, whereas our full model avoids such error. This positive effect is brought by the extraction results preserved in memory module, which serve as a reminder of what information has already been selected. We also note that the ablated model selects wrong sentences in 5-th window. This is because that the model mistakes the  self-esteem  as the salient information. By contrast, our model, being aware of previous texts, correctly captures the  social isolation  as the core topic and filters the noisy sentences. Quantitative analysis . In Figure 8, we compare the Rouge scores between our full model and the ablated one. As can be seen, the performance declines dramatically on both datasets when the memory module is removed. This proves that the dynamic memory indeed plays a necessary role in our model. We further analyze the effecf of memory module in better granularity. Intuitively, the memory module should enhance our model in the following aspects: (1) Reducing Redundancy . Our mem-45.03 19.03 32.58 39.95 16.46 28.19 0 10 20 30 40 50 R-1 R-2 R-L full model w/o memory ArXiv PubMed 46.73 21 34.1 41.46 18.38 30.05 0 10 20 30 40 50 R-1 R-2 R-L full model w/o memory Figure 8: Rouge results of our full model and the ablated version on the two datasets. ory module explicitly records the previous predictions and functions like a sentence-level coverage mechanism, which is expected to reduce repetition. (2) Avoiding Noise . As discussed in Section 1, segment-wise extraction tend to mistake locally important content as summary sentences due to the lack of global context. Our memory module allows the cross-window information flow and therefore should alleviate this problem. (3) Perceiving Sentence Length . The awareness of previous selections may also allow the model to capture sentence length information (Zhong et al., 2019). Ideally, our model is able to adaptively change the change the length of extracted sentence, thereby achieving better performance. formance on above aspects. Similar to (Zhong et al., 2019), we use S Rep = 1  CountUniq ( ngram ) Count ( ngram ) to measure the degree of repetition, where Count ( ngram ) and CountUniq ( ngram ) are the total and unique number of ngrams of selected sentences. For the noise measurement, we have S noise = Count ( NoisySent ) Count ( ExtractSent ) , where NoisySent are the sentences with "R-1" smaller than a threshold. For the length deviation, we have S Len = ( | sum || ref | ) | ref | , where | sum | and | ref | denote the length of model-produced summary and reference summary, respectively. Table 4 presents the comparison results. The model achieves better performance in three indicators when combined with memory mechanism, consistent with aforementioned analysis. In this study, we propose a novel extractive summarization that can summarize long-form documents without content loss. We conduct extensive experiments on two well-studied datasets that consist of scientific papers. Experimental results demonstrate that our model outperforms previous state-of-the-art models. In the future, we will extend our framework (i.e., a sliding encoder combined with long-range memory modeling) to abstractive summarization models.
abstain abstain objective method objective method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method method abstain abstain objective objective abstain objective abstain result method other other other other other other other other other other other other other other other other objective other other other method method abstain abstain method abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other objective abstain method result abstain abstain abstain abstain abstain result abstain method abstain abstain method other abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method other abstain method method abstain method method abstain other method abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result result abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective method objective objective