Neural abstractive summarizers generate summary texts using a language model conditioned on the input source text, and have recently achieved high ROUGE scores on benchmark summarization datasets. We investigate how they achieve this performance with respect to human-written gold-standard abstracts, and whether the systems are able to understand deeper syntactic and semantic structures. We generate a set of contrastive summaries which are perturbed, deficient versions of human-written summaries, and test whether existing neural summarizers score them more highly than the human-written summaries. We analyze their performance on different datasets and find that these systems fail to understand the source text, in a majority of the cases. Open-domain abstractive summarization is a longstanding goal of the field of automatic summarization. Compared to extractive techniques, abstraction offers the potential to generate much more useful summaries by simplifying and rephrasing the source text (Knight and Marcu, 2002), and furthermore by aggregating information and performing operations which are not possible with extractive techniques (Genest and Lapalme, 2012; Carenini and Cheung, 2008). Recently, a number of abstractive summarization systems based on neural sequence-to-sequence architectures have been proposed (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017; Paulus et al., 2018; Chen and Bansal, 2018). These systems learn a compressed representation of the source text using an encoder, then generate the output summary using a conditional decoder. Such neural abstractive systems have achieved very good ROUGE scores on different datasets. Source A former Iraqi army chief of staff being investigated in denmark for war crimes is believed to be back in Iraq , one of his sons said Tuesday. Our interest in this paper is to investigate how these abstractive systems achieve such results, and whether they represent progress towards language understanding and generation. ROUGE arguably provides a limited view of the performance of such systems, as they only relate the system summary to a fixed number of gold-standard summaries. We propose a novel method to directly test the abstractive summarizers in terms of how they score potential candidate summaries, viewing them as conditional language models. This allows us to test whether the summarizers favour output summaries with specific desired qualities, such as generating a summary that is semantically consistent and entailed by the source text. We test how well the neural abstractive summarizers distinguish human-written abstracts from contrastive distractors, which are clearly incorrect summaries that are generated using a rule-based procedure. Table 1 shows contrastive examples which are clearly incorrect 1 . In majority of source texts, we are able to find a contrastive example that scores more highly than the gold-standard summary. 1 For other NLP tasks, others have proposed a similar notion called adversarial examples (Jia and Liang, 2017). Since the term adversarial' has traditionally implied learning to specifically attack the weakness of a model, which we do not do, we refrain from using the word adversarial'. Our work demonstrates the difficulty of controlling expressive neural abstractive systems to produce correct and fluent output. It also underscores the need to revisit fundamental issues in summarization evaluation for neural abstractive models, so that a comprehensive evaluation scheme that captures all relevant aspects of summary quality can be developed. Our code for generating contrastive summaries is available online. 2 2 Related Work Most work in neural abstractive summarization has focused on optimizing ROUGE, whether implicitly by maximum likelihood training or explicitly by reinforcement learning. While this could certainly capture aspects of the content selection problem, we believe that the focus should now shift towards semantic correctness and readability. Cao et al. (2018) took a step in this direction through their fact-aware neural abstractive summarization system. They use fact descriptions of the source as additional features for the summarizer, and showed improved faithfulness according to human judgments. Multi-task learning is another approach used by Pasunuru et al. (2017) to reduce semantic errors in the generated summaries. They jointly learn summarization and entailment generation tasks, using different encoders but a shared decoder. A number of automatic evaluation metrics have shown high correlation with human judges (Liu and Liu, 2008; Graham, 2015), but these results are either restricted to extractive systems or were performed with respect to human-generated summaries. Correlation values are significantly reduced when performed on abstractive summarization systems and datasets (Toutanova et al., 2016). In this section, we describe our method for evaluating summarization systems based on whether they can separate human-written gold summaries from automatically generated contrastive summaries. We define a contrastive summary to be similar to a gold summary, except it contains a perturbation. The perturbation results in either a semantic discrepancy, where facts in source and summary do not corroborate, or a readability issue, where issues with grammar or fluency renders 2 https://github.com/krtin/ ContrastiveSummaries Rule Switching Criteria Noun NN, NNP or NNS must match. the summary clearly incorrect. Below, we first describe our basic method of introducing these discrepancies, then describe a number of restrictions we apply to ensure that the generated contrastive summaries are of high quality. Perturbation by switching words. Given pairs of source texts and gold summaries, we generate multiple contrastive summaries for each source text by perturbing its associated gold summary. There are many types of possible perturbations, but we focus on two strategies: 1) switching words within a gold summary, and 2) replacing a word in gold summary by a word from the source text. We chose these types of perturbations as they are likely to result in difficult contrastive summaries that contain words which are likely to appear in a reasonable summary of the source text, but which are nevertheless incorrect. In order to select the words to be swapped, we apply four rules, separated by syntactic category, as shown in Table 2, using the dependency parse of the texts (Manning et al., 2014). We switch words either within a gold summary, or from the source text and use a single rule at a time for generating a contrastive summary. For example, if the POS tag NNS is matched between a word sides' in the source text and the word combatants' in the gold summary, then the Noun rule would apply, and the words would be switched to obtain the contrastive summary. Further, for the Noun and Verb categories, the switched words may not match in number or verb conjugation. We use SimpleNLG (Gatt and Reiter, 2009) to convert the word to the appropriate inflectional form of the destination's POS tag. Further restrictions. We place a number of restrictions on the words switched, to ensure that the generated summary is contrastive compared to the gold summary. We do not allow switching of the same words. We also do not allow words to be switched if they are separated by any of the following: or' , and' or ,' , as these are likely to be commutative operators. Furthermore, we only allow switching of words from the source text if the context of the words to be switched sufficiently differ from each other. We compute unigram overlap around a context window of size 2 on each side, and allow a switch when the overlap proportion is less than 0.65. These settings were determined by manual inspection of the generated summaries, and allowed us to reduce the number of examples where generated summary is not contrastive, without significantly reducing the number of generated summaries. We will describe the results of a human verifi-cation study in Section 5, in which we ask human raters to check the quality of our contrastive summaries. We apply our set of contrastive summaries to evaluate a number of neural abstractive summarizers. For each summarizer under evaluation, we assume access to a conditional language model which de-fines a probability distribution over words conditioned on the source. Formally, such a language model is given by Equation 1: P ( y i | , S , y 1 ...y i  1 ) , (1) here S is the source,  are model parameters, y i  V sm represents the i th word in the summary, V sm is the vocabulary space of the summary and P is the conditional probability. S  ( s 1 , ..., s n ) where s i  V so , V so is the vocabulary space of the source and n is the maximum source length. Further, we use Equation 2 as our scoring function, p ( y ) = m (cid:88) i =1 log P ( y i | , S , y 1 ...y i  1 ) , (2) here m is maximum summary length and y represents a gold or contrastive summary. For a given triple of source, gold (g) and contrastive (c) summary, if p ( g ) > p ( c ) , then we label the triple dodged' , since the summarizer successfully dodged the generated contrastive summary. If a system is able to dodge all contrastive summaries generated from a source and gold summary tuple, then we label the tuple as escaped' . Datasets. We experimented on two datasets, for two abstractive summarization tasks. The first is a short summarization task, where the summary is one sentence long, for which we use the Gigaword corpus (Graff et al., 2011; Napoles et al., 2012). We use the scripts provided by Rush et al. (2015) to process the Gigaword corpus, which contains the first sentence of the article and the headline as source and gold summary pairs. The test set contains about 250K source-summary pairs from which we randomly selected 10K pairs and generated 509K contrastive summaries. The second is a long summarization task, in which the summary consists of multiple sentences. We use the CNN/Dailymail corpus, where the highlights of the articles are used as the gold summary (Hermann et al., 2015). We used the scripts from Nallapati et al. (2016) to get the data and use the non-anonymized version like See et al. (2017). We use 11.49K test pairs and were able to generate 563K contrastive summaries. Models. We analyze and evaluate three state-of-the-art neural abstractive summarization systems: ABS+ (Rush et al., 2015), GTP (See et al., 2017) and FAS (Chen and Bansal, 2018). The ABS+ system uses an attention-based neural language model as an encoder and a feed-forward neural network for decoding, and is trained on the Gigaword corpus. The GTP system is a seq2seq model with attention on the encoder and a pointer-generator mechanism to choose words from the source in the decoder and, is trained on the CNN/Dailymail corpus. FAS uses reinforcement learning algorithm to extract the most important sentences from the source text, and then summarizes each sentence using a similar architecture as GTP on the CNN/Dailymail corpus. These systems have performed well on small and large text summarization tasks, and have open-source implementation available from the authors. We would have liked to test other relevant systems (Pasunuru et al., 2017; Cao et al., 2018), but were unable to obtain their implementations. contrastive summaries is very large. To restrict the number of contrastive summaries we randomly select approximately 50 generated summaries, while maintaining the rule-wise distribution. The rule-wise distribution was estimated based upon contrastive summaries, generated from a subset of 100 gold standard summaries from CNN/Dailymail corpus. In order to correctly evaluate the FAS system, for each extracted sentence we generate all sentences in the gold summary, and pick the set of summaries with the maximum probability. Human verification. To ensure that we are generating incorrect contrastive summaries, 200 randomly sampled summaries from the Gigaword corpus were evaluated by a human annotator, to verify if a semantic discrepancy or a readability issue was present. We ensured that we sample equally across all the 8 rules, and that we restrict our set of contrastive summaries which the ABS+ system was not able to dodge' . We found that 49.5% had a readability issue while 43.5% had a discrepancy issue, and 93% of the examples had at least one of these issues. This indicates that the vast majority of our contrastive examples are true negatives; i.e., a perfect summarization system should score them lower than the gold standard summary. We summarize our results in Table 4, and report rule-wise results in Table 3. We also include examples where the ABS+ system is unsuccessful in dodging the generated contrastive summaries, in Table 5. Since these metrics directly evaluate the posterior distribution of a summarizer, it al-Model Dataset Dodged Escaped GTP CNN 96.3% 29.8% FAS CNN 92.9% 12.2% ABS+ Gigaword 48.6% 10.8% Table 4: Performance on dodged , escaped metrics. lows us to explicitly recognize problematic examples for a model. We also look at what percentage of gold summaries lie across different ranks of gold summaries in Figure 1. This gives us an insight into distribution of gold summaries, across different ranks. The rank of a gold summary is 1 plus the number of contrastive summaries that scored higher than it. Dodged and Escaped. On the CNN/Dailymail dataset, we find both the models were able to dodge most of the contrastive summaries, but a large number of summaries had at least a few contrastive summaries which scored higher than the gold summary, as reflected by the escaped metric. The FAS model performs worse than GTP model, this might be because the abstraction model only observes one sentence, and thus the probability of observing a word outside the source sentence is higher for the contrastive summaries. Rule-wise Analysis. The GTP and FAS models perform better on rules which switch words within the gold-standard summary. Thus, the decoder LSTM has captured the data distribution very well for words within the summary but is not generalizing for words outside it. This suggest that using words outside the source vocabulary might help in generating harder contrastive examples. The ABS+ model is better in capturing data distributions of prepositions and adjectives. This points towards biases towards particular distributions and can be helpful in further improving these models. Rank of Gold Summary. As shown in Figure 1, almost all the gold summaries have rank lower than 8 for the GTP model, while a large percentage of gold summaries have rank greater than 20, for the ABS+ model. The maximum rank in both the cases is of the order 500K, which is the size of contrastive summaries (Section 5). We suspect that this might be due to behaviourally extractive nature of GTP model, which allows it to easily distinguish any perturbations in the contrastive summaries. We proposed to analyze existing neural abstractive summarizers by testing how they score contrastive summaries, compared to gold-standard ones. For the majority of the gold-standard summaries, we were able to find contrastive examples which score more highly according to current state-of-the-art systems. These examples can be useful not only in evaluating the performance of these systems, but also for improving these systems in the future.
abstain objective method result abstain abstain abstain abstain abstain abstain objective abstain objective method method abstain result abstain method objective abstain method abstain method abstain abstain abstain abstain abstain abstain method method other abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain method method abstain method method abstain abstain method method method abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain