We introduce MemSum (Multi-step Episodic Markov decision process extractive SUMma-rizer), a reinforcement-learning-based extractive summarizer enriched at each step with information on the current extraction history. When MemSum iteratively selects sentences into the summary, it considers a broad information set that would intuitively also be used by humans in this task: 1) the text content of the sentence, 2) the global text context of the rest of the document, and 3) the extraction history consisting of the set of sentences that have already been extracted. With a lightweight architecture, MemSum obtains state-of-the-art test-set performance (ROUGE) in summarizing long documents taken from PubMed, arXiv, and GovReport. Ablation studies demonstrate the importance of local, global, and history information. A human evaluation confirms the high quality and low redundancy of the generated summaries, stemming from MemSum's awareness of extraction history. Automatic text summarization is the task of automatically summarizing a long document into a relatively short text while preserving most of the information (Tas and Kiyani, 2007). Text summarization methods can be categorized into abstractive and extractive summarization (Gambhir and Gupta, 2017; Nenkova and McKeown, 2012). Given a document d consisting of an ordered list of N sentences, extractive summarization aims to pick up M ( M  N ) sentences as the summary of the document. The extracted summaries tend to be both grammatically and semantically more reliable than abstractive summaries (Liu* et al., 2018; Liu and Lapata, 2019a; Luo et al., 2019; Liao et al., 2020), as they are directly selected from the source text. Step Remaining sentences Extracted sentences Select Stop extraction Select Agent Agent Agent Extracted summary Score & Select Action sentence scoring phase, an affinity score is computed for each sentence by neural networks such as bidirectional RNNs (Dong et al., 2018; Narayan et al., 2018; Luo et al., 2019; Xiao and Carenini, 2019) or BERT (Zhang et al., 2019; Liu and Lapata, 2019b). In the sentence selection phase, sentences are selected by either i) predicting a label (1 or 0) for each sentence based on its score, and selecting sentences with label 1 (Zhang et al., 2019; Liu and Lapata, 2019b; Xiao and Carenini, 2019), or ii) ranking sentences based on their scores and selecting the top K sentences as the summary (Narayan et al., 2018), or iii) sequentially sampling sentences without replacement, where the normalized scores of the remaining sentences are used as sampling likelihoods (Dong et al., 2018; Luo et al., 2019). In these approaches, sentence scores are generally not updated based on the current partial summary of previously selected sentences, indicating a lack of knowledge of extraction history . We deem extractive summarizers that are not aware of the extraction history to be susceptible to redundancy in a document, because they will repeatedly add sentences with high scores to a summary, regardless of whether similar sentences have been selected before. And, redundancy leads to performance decreases evaluated by ROUGE F1. In this paper, we propose to model extractive summarization as a multi-step episodic Markov Decision Process (MDP). As shown in Figure 1, at 6507 each time step in an episode, we define a sentence state composed of three sub-states: 1) the local content of the sentence, 2) the global context of the sentence within the document, and 3) information on the extraction history, including the previously selected set of unordered sentences and the remaining sentences. At each time step, the policy network (agent) takes the current sentence state as input and produces scores used to select an action of either stopping the extraction process or selecting one of the remaining sentences into the candidate summary. Unlike one-step episodic MDP-based models (Narayan et al., 2018; Dong et al., 2018; Luo et al., 2019) that encode the state information only once at the beginning of the episode, in our multi-step policy, the agent updates at each time step the extraction history before selecting an action. Such a step-wise state-updating strategy enables the agent to consider the content of the partial summary when selecting a sentence. To efficiently encode local and global sentence states, we design an extraction agent based on LSTM networks (Hochreiter and Schmidhuber, 1997). To encode the extraction history and to select actions, we use a reduced number of attention layers (Vaswani et al., 2017) of relatively low dimensionality. These choices enable our model to be easily trainable and to summarize long documents such as scientific papers (Cohan et al., 2018; Huang et al., 2021) or reports (Huang et al., 2021). The contributions of our work are as follows: 1) We propose to treat extractive summarization as a multi-step episodic MDP that is aware of the extraction history. 2) We show that extraction-history awareness allows our model to extract more compact summaries than models without history awareness and behave more robustly to redundancies in documents. 3) Our model outperforms both extractive and abstractive summarization models on PubMed, arXiv (Cohan et al., 2018), and GovReport (Huang et al., 2021) datasets. 4) Finally, human evaluators rate the MemSum summaries to be of higher quality than those from a competitive approach, especially by virtue of lower redundancy 1 . Extraction history awareness was previously considered in NeuSum (Zhou et al., 2018), where a GRU encoded previously selected sentences into a hidden vector that then was used to update the scores of the remaining sentences to bias the next selection. NeuSum contains no stopping mechanism and therefore it can only extract a fixed number of sentences, which likely is sub-optimal. Also, the potential benefits of extraction history have not been quantified and so the idea remains unexplored to a large extent. Recently, BERT-based extractors such as MatchSum (Zhong et al., 2020) achieved SOTA performance in extractive summarization of relatively short documents from the CNN/DM (Hermann et al., 2015) dataset. However, the quadratic computational and memory complexities (Huang et al., 2021) of such models limit their scalability for summarizing long documents with thousands of tokens, which is common for scientific papers and government reports. Although large pre-trained transformers with efficient attention (Huang et al., 2021) have been adapted for abstractive summarization of long documents, we believe that extractive summarization is more faithful in general, which is why we chose an extractive approach. This section outlines the multi-step episodic MDP policy for extractive summarization. In an episodic task with a terminal state (i.e. end of summary ), policy gradient methods aim to maximize the objective function J (  ) = E   [ R 0 ] , where the return R t = (cid:80) Tk = t +1 r k is the cumulative reward from time t + 1 until the end of the episode when the summary is complete. In applications of RL to extractive summarization, the instantaneous reward r t is zero except at the end of the episode when the final reward r is computed according to Equation (1), so R t  R 0 = r . The reward r is usually expressed as (Dong et al., 2018): r = 1 3( ROUGE-1 f + ROUGE-2 f + ROUGE-L f ) (1) According to the REINFORCE algorithm (Williams, 1992), the policy gradient is defined as:  J (  ) = E  [ R t  log  ( A t | S t ,  )] , (2) where  ( A t | S t ,  ) denotes the likelihood that at time step t the policy   selects action A t given the 6508 Update extracted sentences with LSE MHP words of sentence GCE All sentences' embeddings Extracted sentences Remaining sentences sample actions Sampled actions stop? state S t . With  as the learning rate, the parameter update rule is (Sutton and Barto, 2018):  t +1   t + R t  log  ( A t | S t ,  t ) , (3) 3.2 Multi-step Episodic MDP Policy Different from one-step episodic MDP policies (Narayan et al., 2018; Dong et al., 2018; Luo et al., 2019) that extract the entire summary via a single action, we define an episode, i.e., the generation of a summary, consisting of multiple time steps. At each time step t , corresponding to extracting sentence number t , the action A t is either to stop extraction or to select a sentence s a t from the remaining sentences. The agent's policy is:  ( A t | S t ,  t ) = p ( stop | S t ,  t ) p ( a t | stop , S t ,  t ) p ( a t | stop , S t ,  t ) =   u at ( S t ,  t ) (cid:80) j  It u j ( S t ,  t ) if stop = false 1 | I t | if stop = true , (4) where I t denotes the index set of remaining sentences at time step t . If the agent does not stop, it first computes a score u j for each remaining sentence and samples a sentence s a t according to the probability distribution of normalized scores. When the agent stops the extraction, no sentence is selected and the conditional likelihood p ( a t | stop=false , S t ,  t ) is set to 1 | I t | (where | I t | represents the number of remaining sentences at time t ), which is independent of the policy parameters to prohibit the gradient from being passed to the policy parameters via the conditional likelihood. After calculating the reward according to Equation (1), the policy parameters are updated according to Equation (3) (for all time steps). The state S t in Equation (4) is designed to be informative on: 1) the local content of the sentence, 2) the global context of the sentence within the document, and 3) the current extraction history. To encode these three properties in the state, we use a local sentence encoder, a global context encoder, and an extraction history encoder, respectively. Subsequently, the state is mapped by an extractor to an output score for each of the remaining sentences and the extraction stop signal. The overall framework of our model is depicted in Figure 2. In the Local Sentence Encoder (LSE), ordered words ( w 1 , w 2 , . . . w M ) in a sentence s i are first mapped onto word embeddings using a word embedding matrix. Subsequently, a N l -layer bidirectional LSTM (Hochreiter and Schmidhuber, 1997) transforms the word embeddings and maps them onto sentence embeddings l s i via a multihead pooling layer (MHP) (Liu and Lapata, 2019a). The Global Context Encoder (GCE) consists of a N g -layer bi-LSTM that takes the L local sentence embeddings ( l s 1 , l s 2 , . . . l s L ) as inputs and produces for each sentence s i an embedding g s i that encodes global contextual information such as the sentence's position in the document and information on neighboring sentences. The Extraction History Encoder (EHE) encodes the extraction history information and produces the extraction history embedding h s ri for each remaining sentence s ri . The EHE is composed of a stack of N h identical layers. Within one layer, there are two multi-head attention sublayers, as contained in the transformer decoder in Vaswani et al. (2017). One sublayer is used to perform multi-head self-attention (MHA) among the local embeddings 6509 of the remaining sentences, so that each remaining sentence can capture the context provided by other remaining sentences. The other attention sublayer is used to perform multi-head attention over the embeddings of extracted sentences to enable each remaining sentence to attend to all the extracted sentences. The output of the two attention sublayers, one for each remaining sentence, captures the contextual information of both extracted and remaining sentences. The final output of the N th h layer of the EHE constitutes the extraction history embedding, one for each remaining sentence. There is no positional encoding and the EHE produces the extraction history embeddings non-autoregressively by attending to both precedent and subsequent positions. Consequently, the extraction history embeddings h s ri for the remaining sentences are invariant to the order of the previously selected sentences. We believe that the sequential information of previously selected sentences is not crucial for reducing redundancy and for deciding whether to stop extraction or not. The Extractor computes the score of each remaining sentence and outputs an extraction stop signal. As input to the extractor, we form for each of the remaining sentences s ri an aggregated embedding by concatenating the local sentence embedding l s ri , the global context embedding g s ri , and the extraction history embedding h s ri . As shown in Figure 2, to produce the score u s ri , the concatenated embedding of remaining sentence s ri is passed to fully connected layers with ReLU activation and then projected to a scalar by a Linear-1 layer followed by a sigmoid function. Note that the same fully connected layers are applied identically to all remaining sentences. We deem that the extractor can learn to stop extraction based on the remaining sentences' states. Therefore, we apply an MHP to the last hidden vectors of all remaining sentences to output a single vector. This vector is then passed to a linear layer with a sigmoid function, producing a stopping probability p stop . We train the parameterized policy network according to the update rule in Equation (3). At each training iteration, an episode is sampled to compute the final return r and the action probabilities  ( A t | S t ,  t ) for all time steps t . An example episode with T extracted sentences looks like: ( S 0 , s a 0 , . . . , ST  1 , s a T  1 , ST , A stop , r ) , where S t represents the concatenated state information introduced in Section 3.3, s a t represents the selection of sentence a t , A stop represents the extraction stops at the final time step T , and r is the reward as defined in Equation (1). To encourage the agent to select compact summaries, we multiply the final reward r by a length penalty term 1 / ( T + 1) (Luo et al., 2019). Consequently, the return R t  r T +1 . 1: for each document-summary pair ( D i , G i ) do 2: LSE outputs local sent. embed l s 1 , . . . , l s L 3: GCE outputs global context embed g s 1 , . . . , g s L 4: Sample an episode S 0 , s a 0 , . . . , ST  1 , s a T  1 , ST , A stop , r from the high-ROUGE episodes set E p of document D i 5: for each time step: t = 0,1,...,T: do 6: if t > 0 then 7: EHE outputs extraction history embed h s r 1 , . . . , h s rL  Et for remaining sentences 8: else 9: Initialize h s r 1 ,..., h s rL  E 0 to 0 10: Extractor outputs scores u s r 1 ,..., u s rL  Et for remaining sentences and outputs p stop 11: Compute the action probability  ( A t | S t ,  ) according to Equation (4) 12:    +  r T +1  log  ( A t | S t ,  ) Algorithm 1 summarizes the training procedure of MemSum. We initialize the extraction history embeddings to 0 , because at t = 0 no sentences have been extracted. E t represents the number of sentences that have been extracted into the summary up to time step t . Following the strategy in Narayan et al. (2018) and Mohsen et al. (2020), instead of sampling an episode following the current policy  ( | ,  t ) , we sample an episode from a set E p of episodes with high ROUGE scores, which enables the agent to quickly learn from optimal policies and to rapidly converge. Details on creating a set of high-ROUGE episodes for training are described in Appendix E. 4 Experiments In this section, we report implementation details of our model and describe the datasets used for training and for evaluation. Datasets. The documents to be summarized in the PubMed and arXiv datasets (Cohan et al., 2018) 6510 Datasets avg. are the full bodies of scientific papers and the gold summaries are the corresponding abstracts. Zhong et al. (2020) proposed a truncated version of the PubMed dataset (PubMed trunc for simplicity) by defining a doument as the introduction section of a paper. The GovReport dataset (Huang et al., 2021) contains U.S. government reports with gold summaries written by experts. Except PubMed trunc , all the other datasets contain significantly longer documents than the popular dataset CNN/DM (Table 1). Baselines. Extractive baselines include Lead (directly using the first several sentences as the summary) (Gidiotis and Tsoumakas, 2020), SummaRuNNer (Nallapati et al., 2017), Atten-Cont (Xiao and Carenini, 2019), Sent-CLF and Sent-PTR (Pilault et al., 2020), MatchSum (Zhong et al., 2020), and the NeuSum model (Zhou et al., 2018) that we trained on our datasets. Abstractive summarization models include PEGASUS (Zhang et al., 2020), BigBird (Zaheer et al., 2020), Dancer (Gidiotis and Tsoumakas, 2020), and Hepos (Huang et al., 2021) that achieved the state-of-the-art in long document summarization using a large-scale pretrained BART model (Lewis et al., 2020) with memory-efficient attention encoding schemes including Locality Sensitive Hashing (Kitaev et al., 2020) (Hepos-LSH) and Sinkhorn attention (Hepos-Sinkhorn). We also present the performance of the oracle extraction model based on the greedy approach (Nallapati et al., 2017) which sequentially selects from the document the sentence that maximally improves the average of R-1 and R-2 of selected sentences. Implementation Details. We computed local sentence embeddings using pretrained Glove word embeddings (Pennington et al., 2014) of dimension d = 200 , keeping the word embeddings fixed during training. For the LSE, we used N l = 2 bi-LSTM layers and for the GCE N g = 2 . For the EHE, we used N h = 3 attention layers, and we set the number of attention heads to 8 and the dimension of the feed-forward hidden layer to 1024 ; during training we set the dropout rate to 0 . 1 . The extractor consisted of 2 fully-connected hidden layers with output dimensions 2 d and d , respectively. We trained our model using the Adam optimizer with  1 = 0 . 9 ,  2 = 0 . 999 (Kingma and Ba, 2015), fixed learning rate  = 1 e  4 , and weight decay 1 e  6 . The training was stopped when the validation performance started to degrade. During validating and testing, the agent extracted sentences in a deterministic way: after computing the scores u s ri for the remaining sentences and the stop likelihood p stop , the agent stopped the extraction if p stop  p thres or if the maximum admissible number N max of extracted sentences was reached; otherwise, the agent selected the sentence with the largest score. The model was trained on eight RTX 2080 Ti GPUs. On the validating datasets we selected the best checkpoint of each model and determined the optimal N max and stopping criterion p  thres . For Pubmed, arXiv, Pubmed trunc , and GovReport, N max was set to 7 , 5 , 7 , and 22 , and p  thres was set to 0 . 6 , 0 . 5 , 0 . 8 , and 0 . 6 , respectively. For the detailed selection procedure of the optimal stopping threshold, see Appendix D. Information on reproducibility is available in Appendix I. Evaluation. We evaluated the performance of our model using F 1 ROUGE (Lin, 2004), including ROUGE-1,2, and L for measuring unigram, bigram, and longest common subsequence. We also conducted human evaluation in Section 5.4. Here we present the results on various extractive summarization tasks and analyze the contribution of different modules via ablation studies. By comparing with extractive baselines on the PubMed and arXiv datasets, we observed that models utilizing extraction history, such as NeuSum and our MemSum, perform significantly better than other models, revealing the effectiveness of the extraction history. MemSum also significantly outperformed NeuSum, suggesting a better utilization of extraction history , which we ascribed to the following factors: 1) In MemSum, we treat stopping extraction also as an action and train the policy network to output a stopping probability. There-6511 Model PubMed arXiv R-1 R-2 R-L R-1 R-2 R-L ORACLE 61.99 34.95 56.76 60.00 30.60 53.03 Extractive summarization baselines Lead-10 37.45 14.19 34.07 35.52 10.33 31.44 SummaRuNNer 43.89 18.78 30.36 42.81 16.52 28.23 Atten-Cont 44.85 19.70 31.43 43.62 17.36 29.14 Sent-CLF 45.01 19.91 41.16 34.01 8.71 30.41 Sent-PTR 43.30 17.92 39.47 42.32 15.63 38.06 NeuSum 47.46 21.92 42.87 47.49 21.56 41.58 Abstractive summarization baselines PEGASUS 45.97 20.15 41.34 44.21 16.95 38.83 BigBird 46.32 20.65 42.33 46.63 19.02 41.77 Dancer 46.34 19.97 42.42 45.01 17.60 40.56 Hepos-Sinkhorn 47.93 20.74 42.58 47.87 20.00 41.50 Hepos-LSH 48.12 21.06 42.72 48.24 20.26 41.78 MemSum (ours) 49.25 * 22.94 * 44.42 * 48.42 20.30 42.54 * Table 2: Results on the PubMed and arXiv test sets. fore, MemSum is able to automatically stop extracting at an optimal time step based on extraction history, while NeuSum can only extract a predefined number of sentences; 2) With the policy gradient method REINFORCE we can train MemSum to maximize the ROUGE score directly, while in NeuSum the loss was set to the KL-divergence between the model-computed sentence scores and the ROUGE score gains at each step, which is less intuitive. We further compare MemSum with NeuSum via human evaluation in Section 5.4. We observed that the ROUGE performance on the PubMed trunc dataset is significantly lower than that on the PubMed dataset, with a 16.87 drop in R-1 for the extractive oracle and a 6.23 drop in R-1 for MemSum, indicating that the introduction section is not sufficient to generate summaries close to the ground truth (abstracts). Even so, our model still significantly outperformed MatchSum on PubMed trunc , and we attribute this improvement to the fact that MatchSum truncates the introduc-2 https://pypi.org/project/rouge-score/ 0 5 10 15 20 25 30 Extracted Sentence Position 0 10 % o f s e n t e n c e p o s i t i o n ORACLE MEMSUM (ours) MatchSum Figure 3: The position distribution of extracted sentences in the PubMed trunc dataset. Human-written Summary: (...) While CMS is generally required to disallow, or recoup, federal funds from states for eligibility-related improper payments if the state's eligibility error rate exceeds 3 percent , it has not done so for decades, (...) CMS issued revised procedures through which it can recoup funds for eligibility errors, beginning in fiscal year 2022 . (...) Hepos-Sinkhorn (abstractive): (...) The selected states also reported that they did not have adequate processes to address these issues. CMS has taken steps to improve its oversight of the Medicaid program, including issuing guidance to states on the use of MAGI-exempt bases for determining eligibility, but these efforts have not been fully implemented. (...) MemSum (ours, extractive): (...) implemented its statutory requirement to recoup funds associated with Medicaid eligibility-related improper payments for states with an eligibility error rate above 3 percent through its MEQC program. (...) However, the agency has introduced new procedures through which it can, under certain circumstances, begin to recoup funds based on eligibility errors in fiscal year 2022 . (...) tion section further to 512 tokens because it needs to compute document embeddings using Bert. Consequently, MatchSum extracts sentences mainly from the first 15 sentences of the document, while our MemSum produces a similar distribution of extracted sentence positions as the extractive oracle, Figure 3. Thus, summarizing long documents is a non-trivial task , and models that work well on summarizing short documents (e.g., CNN/DM) may fail to generalize to long documents. MemSum also significantly outperformed the state-of-the-art abstractive summarization model Hepos as measured by ROUGE scores, especially on the GovReport dataset. A comparison of an exemplary MemSum-extracted summary and the corresponding Hepos-Sinkhorn-generated summary from the GovReport dataset (Table 4) is consistent with the ROUGE comparison, showing that the MemSum-extracted summary is more accurate 6512 Model R-1 R-2 R-L MemSum 49.25 22.94 44.42 MemSum w/o LSE 48.12 22.04 43.36 MemSum w/o GCE 46.85 20.31 41.95 MemSum w/o EHE 48.08 22.77 43.55 MemSum with GRU-EHE 49.11 22.86 44.28 MemSum w/o auto-stop 48.25 22.63 43.70 MemSum with STOP 47.18 21.81 42.20 Table 5: Ablation study on the PubMed dataset. than the Hepos-Sinkhorn-generated summary and has higher overlap with the gold summary. We deem that this particularly good extraction performance on the GovReport dataset results from the higher extractiveness of the gold summaries in the GovReport dataset compared to other datasets, which may be due in part to technical language being difficult to abstractively summarize without a change in meaning. This is evidenced by the fact that the ROUGE scores of the extractive oracle on the GovReport dataset (Table 3) are higher than those of the PubMed and arXiv datasets (Table 2). Therefore, extractive summarization may be more proper than abstractive summarization due to the requirement of stringent faithfulness of government report summaries . We conduct ablation studies by comparing the full MemSum model with the following variations in structures : 1) MemSum w/o LSE, where we obtain local sentence embeddings by replacing the bi-LSTM based LSE by simple averages of word embeddings; 2) MemSum w/o GCE where we remove the GCE; 3) MemSum w/o EHE where we remove EHE, compute the scores for all sentences in one step, and samples sentences following the BanditSum policy (Dong et al., 2018); 4) MemSum with GRU-EHE where we use a GRU to encode previously extracted sentences at each time step, and uses the last hidden state as the extraction history embedding for all remaining sentences, following Zhou et al. (2018). Meanwhile, we also tested two variations that adopted different stopping mechanisms : 1) MemSum w/o auto-stop that does not stop extraction automatically based on p stop , but that extracts a fixed number of sentences; 2) MemSum with STOP that inserts a special stop sentence (e.g. STOP") into the document, and stops extraction once the agent selects this sentence. Contribution of Modules. Removing GCE has Model R-1 R-2 R-L duplicatepercentage MemSum 49.16 22.78 44.39 0% MemSum w/o auto-stop 48.21 22.59 43.76 0% MemSum w/o EHE 42.82 18.18 36.68 41% MemSum w/o EHE +3gram blocking 46.85 19.93 42.40 0% Table 6: Performance on the redundant PubMed dataset. a greater impact on performance than removing LSE (Table 5), suggesting that modeling global contextual information is more critical than modeling local sentence information in our MemSum framework, which contrasts with the result that modeling local sentence information is more important in the Atten-Cont (Xiao and Carenini, 2019) framework. Furthermore, we observed a significant performance degradation when removing EHE, but no significant difference between MemSum and MemSum with GRU-EHE, indicating that EHE is necessary, but our MemSum policy is not strongly dependent on the specific structure of this module (e.g., attention-based or RNN-based). Influence of Stopping Mechanisms. MemSum w/o auto-stop achieves lower ROUGE scores than MemSum, revealing the necessity of auto stopping in our MemSum architecture. Meanwhile, MemSum with STOP produced summaries with fewer extracted sentences (3.9 vs. 6.0 sentences on average) and significantly lower ROUGE scores. We attribute this reduction to the predictable positive reward obtained from selecting the special stop sentence that ends an episode, which leads to a preference for this final action and increases the likelihood of taking this action prematurely. We hypothesized that the extraction history allows MemSum to avoid sentences that are similar to existing sentences in the current partial summary, intuitively mimicking what humans do when extractively summarizing documents. To verify this, we created a redundant PubMed dataset in which we repeated each sentence in the document, with the replicated sentences immediately following the originals. On this dataset, we trained and tested MemSum and MemSum w/o EHE (no history awareness), and we compared different models in terms of ROUGE scores and average duplicate percentage that is defined as the average percentage of the duplicated sentences among all extracted sentences in a summary. As reported in Table 6, for MemSum w/o EHE, on average 41% of sentences in the extracted summaries were duplicated. Along with the high duplicate ratio came a significant decrease in ROUGE score. By contrast, the performance of the full MemSum model with history awareness was only slighted affected when comparing the results of the MemSum on the PubMed dataset (Table 2) and on the redundant PubMed dataset (Table 6). Meanwhile, using the Trigram Blocking method that skips a sentence if it has a trigram that overlaps with the current summary (Liu and Lapata, 2019b) is also successful in avoiding repetitive sentences. However, the ROUGE scores associated with Trigram Blocking were significantly lower than those of the MemSum with awareness of extraction history. In summary, the history-aware MemSum model spontaneously learns an optimized strategy to avoid redundant sentences without explicit human guidance or crude rules, and thus shows better performance. Case Study: How does MemSum Avoid Redundancy? We let MemSum summarize a document sampled from the test set of the redundant PubMed dataset and monitored the sentence scores produced by the Extractor during each extraction step. The results are shown in Figure 4. At time step 0 , the 10 th sentence obtained the maximum score and was thus selected into the summary. At time step 1 , we noticed that the 11 th sentence, which is a replica of the 10 th sentence, had a score close to zero. The same was also true for the other selected sentences and their following sentences, revealing competent Criteria Experiment I Experiment II NeuSum MemSum NeuSum MemSum w/o auto-stop overall 1.58 1.37 1.57 1.38 coverage 1.46 1.49 1.44 1.51 non-redundancy 1.67 1.28 * 1.65 1.30 * avg. repetition avoidance of the Extractor. Because the EHE is insensitive to the extraction order and to sentence position information, as described in Section 3.3, we can conclude that the full MemSum avoids redundancy by evaluating the similarity between selected and remaining sentences, rather than by remembering" selected sentences' positions. We conducted human evaluation following Wu and Hu (2018); Dong et al. (2018); Luo et al. (2019). For each document sampled from the test set of the PubMed dataset, we provide a reference summary, and volunteers are asked to rank a pair of randomly ordered summaries produced by two models according to three criteria: non-redundancy, coverage, and overall quality. The better model will be ranked #1 while the other is ranked #2, and if both models extract the same summary, then they will both get the #1 rank. In experiment 1, we compared NeuSum, which always extracts 7 sentences, and MemSum, which extracts a flexible number of sentences thanks to automatic stopping. In experiment 2, we discounted for differences in the number of extracted sentences by making MemSum w/o auto-stop to also extract 7 sentences. A user-friendly interactive web interface was implemented to assist the evaluation process, with details in Appendix G. Table 7 reports the human evaluation results for both experiments. Both MemSum and MemSum w/o auto-stop ranked significantly higher (p<0.005) than NeuSum in terms of non-redundancy and achieved a better average overall quality. In terms of word count, MemSum produces shorter summaries than NeuSum in both experiments, even though both models extract the same number of 6514 sentences in experiment 2. These results show that redundancy avoidance of MemSum is particularly good, even without the auto-stop mechanism. The slightly better performance of NeuSum in terms of coverage needs to be weighed against it extracting significantly longer summaries. Note that neither NeuSum nor our model is trained to optimize the order of the extracted sentences. Therefore, we did not use fluency, which depends on sentence order, as a metric for human evaluation. Improving the fluency of the extracted summaries will be the subject of our future research. Extractive summarization can be achieved effectively with a multi-step episodic Markov decision process with history awareness. Using encoders of local sentence, global context, and extraction history, MemSum is given information that is intuitively also used by humans when they summarize a document. Awareness of the extraction history helps MemSum to produce compact summaries and to be robust against redundancy in the document. As a lightweight model (Appendix C), MemSum outperforms both extractive and abstractive baselines on diverse long document summarization tasks. Because MemSum achieves SOTA performance on these tasks, MDP approaches will be promising design choices for further research. We acknowledge support from the Swiss National Science Foundation (grant 31003A_182638) and the NCCR Evolving Language, Swiss National Science Foundation Agreement No. 51NF40_180888. We also thank the anonymous reviewers for their useful comments.
abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective result abstain method abstain method method abstain objective result other abstain other other other other other other method abstain abstain abstain other abstain method abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain method abstain other method abstain abstain other method abstain method method abstain other abstain abstain abstain abstain abstain other other abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method abstain other result other other abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain other abstain result other result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other other