Generic summaries try to cover an entire document and query-based summaries try to answer document-specific questions. But real users' needs often fall in between these extremes and correspond to aspects, high-level topics discussed among similar types of documents. In this paper, we collect a dataset of realistic aspect-oriented summaries, ASPECTNEWS , which covers different subtopics about articles in news sub-domains. We annotate data across two domains of articles, earthquakes and fraud investigations, where each article is annotated with two distinct summaries focusing on different aspects for each domain. A system producing a single generic summary cannot concisely satisfy both aspects. Our focus in evaluation is how well existing techniques can generalize to these domains without seeing in-domain training data, so we turn to techniques to construct synthetic training data that have been used in query-focused summarization work. We compare several training schemes that differ in how strongly keywords are used and how oracle summaries are extracted. Our evaluation shows that our final approach yields (a) focused summaries, better than those from a generic summarization system or from keyword matching; (b) a system sensitive to the choice of keywords. 1 1 Introduction Recent progress in text summarization (See et al., 2017; Liu and Lapata, 2019; Zhang et al., 2020a; Lewis et al., 2020) has been supported by the availability of large amounts of supervised data, such as the CNN/Daily Mail and XSum datasets (Hermann et al., 2015; Narayan et al., 2018), which provide a single, generic, topic-agnostic summary. However, a document often contains different aspects (Titov and McDonald, 2008; Woodsend and Lapata, 2012) that might be relevant to different users. For 1 Code is available at https://github.com/oja/ aosumm example, a political science researcher studying responses to earthquakes may want a summary with information about government-led recovery efforts and broader social impacts, not a high-level generic summary of what happened. Systems should be able to produce summaries tailored to the diverse information needs of different users. Crucially, these systems should be usable in realistic settings where a user is interested in vague aspects of the document, instead of a highly focused query. In this work, we present a new dataset for evaluating single-document aspect-oriented extractive summarization which we call ASPECTNEWS . We derive subsets of examples from CNN/Daily Mail following certain topics, namely earthquakes and fraud reports. These domains are special in that the articles within them have several aspects which are repeatedly mentioned across articles and form coherent topics, e.g., impact on human lives of an earthquake. We ask annotators to select sentences relevant to such information needs, which correspond to imagined use cases. Interannotator agreement on full summaries is low due to the inherent subjectivity of the task, so rather than coming up with a consensus summary, we instead primarily evaluate against soft labels based on the fraction of annotators selecting a given sentence. To benchmark performance on this dataset, we build a system that can summarize a document conditioned on certain aspect-level keywords without assuming annotated training data for those aspects. Since there are no large-scale supervised training sets suitable for this purpose, we explore methods to generate aspect-oriented training data from generic summaries. We compare these with past approaches (Frermann and Klementiev, 2019) on their ability to adapt to our aspect-oriented setting, which requires taking aspectual keyword inputs (as opposed to specific entities or queries) and being appropriately sensitive to these keywords. 1. At least 42 people have died with hundreds more injured after a 6.2-magnitude earthquake hit Indonesia's Sulawesi island early Friday, according to Indonesia's Disaster Management Agency. 2. The epicenter of the quake, which struck at 1:28 a.m. Jakarta time, was 6 kilometers (3.7 miles) northeast of the city of Majene, at a depth of 10 kilometers (6.2 miles), according to Indonesia's Meteorology, Climatology and Geophysics Agency. 3. Thirty-four people died in the city of Mamuju, to the north of the epicenter, while another eight died in Majene. 4. In Majene, at least 637 were injured and 15,000 residents have been displaced, according to [] 7. Many people are still trapped under collapsed buildings, according to local search and rescue teams. 8. Rescuers search for survivors at a collapsed building in Mamuju city in Indonesia. 9. Our priority is saving victims who are still buried under the buildings," Safaruddin Sanusi, head of West Sulawesi's Communications and Information Department, told CNN Friday. [] 12. Mostof the people in Mamuju city are now displaced. They are afraid to stay at their houses. 15. We need more extrication equipment and more personnel to work fast on saving victims trapped under the building. Generic Geo Recovery 1    2  3   4  7 8  9  12 15 Figure 1: Examples of an earthquake-related article paired with extractive summaries from the CNN/DM dataset. and the SPACE dataset (Angelidis et al., 2021) find that our model produces summaries that score higher on agreement with human aspect-oriented annotations than generic summarization models, previous aspect-oriented models, and baselines such as keyword matching. Second, we find that the summaries our model generates are sensitive to the choice of keywords. Third, we find that our model performs competitively with leading models on the SPACE dataset in the multi-document setting. Finally, we find that abstractive query-focused systems (He et al., 2020) hallucinate significantly in this setting, justifying our choice of an extractive framework here. Relatively little recent work has focused on aspect-oriented summarization. One line of research focuses on summarization of documents with respect to specific queries (Baumel et al., 2014; Krishna and Srinivasan, 2018; Frermann and Klementiev, 2019; He et al., 2020; Xu and Lapata, 2020a). However, a query such as  What facilities were damaged in the Oaxacan region?  is a document specific query, which cannot be applied to other earthquake news articles and bears more resemblance to the task of long-form question answering (Fan et al., 2019). Our focus is closer to work on attribute extraction from opinions or reviews (Dong et al., 2017; Angelidis and Lapata, 2018), as factors like geographic details and recovery efforts are usually mentioned in many earthquake stories. Recent work has also begun to study summarization from an interactive perspective (Shapira et al., 2021); our approach could be naturally extended in this direction. Methods Historically, most work on query-focused summarization has addressed the multi-document setting. You et al. (2011) apply regression models to this task, and Wei et al. (2008) approach the problem from the perspective of ranking sentences by their similarity to the query. These classic methods rely integrally on the multi-document setting, and so cannot be easily adapted to our setup. More recently, Xu and Lapata (2020b) focus on multi-document summarization by modeling the applicability of candidate spans to both the query and their suitability in a summary. Angelidis et al. (2021) explore a method using quantized transformers for aspect-oriented summarization, which we compare to. Datasets There are several differences between ASPECTNEWS and other existing aspect-oriented summarization datasets. Firstly, ASPECTNEWS focuses on single-document summarization, while similar aspect-oriented datasets such as the SPACE dataset of reviews (Angelidis et al., 2021) and other attribute extraction settings (Dong et al., 2017; Angelidis and Lapata, 2018) are multi-document. Second, our dataset focuses on generalization to new aspect types , rather than assuming we've trained on data with those same aspects; that is, how can we produce appropriate aspect-oriented summaries of earthquake articles even if we have not trained on any? Third, compared to query-focused settings, our aspect-oriented dataset is closer to the actual information needs of users, since users are often interested in summaries about broad subtopics rather than specific queries. 2 https://tac.nist.gov/2011/ Summarization propose guided summarization tasks that involve similar aspects. However, each article cluster in TAC has a single, fixed set of aspects that don't differ substantially from what a generic summary should capture. The DUC 2005/2006 task (Dang, 2005) does not have aspects but rather can accept a granularity level at which to produce the summary. Christensen et al. (2014) produce a hierarchy of relatively short summaries among multiple documents. Other previous work (He et al., 2020; Xu and Lapata, 2020a; Tan et al., 2020) proposes constructing keyword sets for each individual document for training. Krishna and Srinivasan (2018); Frermann and Klementiev (2019) condition on topic tokens referring to the topic tags in metadata. Compared to these other approaches, we focus more on evaluation of aspects, as opposed to a purely keyword-and query-driven view. We begin by considering our target application: users who have specific information needs that they want to be satisfied. This consideration broadly falls under the category of purpose factors defined by Jones (1998) and should be accounted for in the summarization process. Our data collection process involves the following steps: (1) Identifying clusters of articles in our target domains from a large corpus of news summaries. (2) Manually specifying multiple user intents per target domain, representing the aspect of the summarization process. (3) Crowdsourcing annotation of extractive summaries in these domains based on the user intents. We draw our datasets from the English-language CNN/Daily Mail summarization dataset (Hermann et al., 2015). We manually identified two domains, earthquakes and fraud , based on inspecting clusters of articles in these domains. These two domains are ideal for two reasons. First, they contain a significant number of on-topic articles (over 200) after careful filtering. Second, the articles in these domains are reasonably homogeneous: each article would often feature at least broadly similar information about an event, making aspect-based summarization well-defined in these cases. 3 Although not completely universal, most earthquake articles refer to some information about each of two aspects here: geography (GEO ) and recovery (RECV ). Figure 1 shows an example of an earthquake-related article. Similarly, most fraud articles include information about the penalty (PEN ) imposed for the fraud, and the nature (NATURE ) of the fraud. To retrieve our examples from these two domains, we first encode each article in CNN/DM corpus C with a text encoder E . We adopt the Universal Sentence Encoder (Cer et al., 2018) for its efficiency and robustness. We create an exemplar sentence for each domain to serve as the target to retrieve the most relevant content. We describe the choice of exemplar sentences in Section A.2. We measure the similarity of each candidate article c and the exemplar sentence s as the average of the cosine similarity between each of the candidate article's sentences c i and the exemplar, sim ( c, s ) = 1 n (cid:80) n i =1 cos( E ( c i ) , E ( s )) . We found this procedure to be more robust than simple keyword matching for retrieving articles with coherent aspects; for example, keyword matching for earthquakes resulted in returning articles primarily about tsunamis due to the imbalanced data distribution. 3 By contrast, other domains like legislation were too heterogeneous: articles about passing a bill may focus on different aspects of a bill's journey, comments or quotes by elected officials, impact of the legislation, or other factors. We could not come up with a plausible unified information need for the sorts of articles available in this dataset, although our eventual system can be applied to such documents if given appropriate guidance. With these two domains, we examine our dataset to derive aspects that simulate realistic information needs of users. Table 1 describes the domain, aspect, annotation prompt and keywords used for evaluation. For each domain, we establish two aspects. Each aspect must be well-represented in the corpus and easy to understand by both readers and annotators. The authors annotated these aspects based on inspection of the articles and brainstorming about user intents based on scenarios. For example, the penalty scenario was motivated by a real use case derived from the authors' colleagues investigating reporting of wrongdoing in news articles at scale, where summarization can be used to triage information. Finally, to construct actual extractive summaries for evaluation in these domains, we presented the user intents to annotators on Amazon Mechanical Turk. An annotator is shown a description of intent from Table 1 along with an article and is asked to identify a few sentences from the article that constitute a summary. They can rate each sentence on a scale from 0 to 3 to account for some sentences being more relevant than others. Their final summary, which they are shown to confirm before submitting, consists of all sentences rated with a score of at least 1. The exact prompt is shown in the Appendix. Each article was truncated to 10 sentences for ease of annotation. This assumption was reasonable for the two domains we considered, and the truncation approach has been used in See et al. (2017) without much performance degradation. We found that annotators were unlikely to read a full length article due to the inherent lead bias in news articles, so this also helped simplify the task. In order to maintain a high quality of annotations, we discard annotations that do not have at least a single selected sentence in common with at least a single other annotator on that sample. In practice, this only discards a handful of isolated annotations. In Table 2, we show the basic statistics of the collected dataset. We show the distribution of the number of sentences agreed upon by the annotators in Table 3. We see that annotators somewhat agree in most cases, but relatively few sentences are uniformly agreed upon by all annotators. Our initial # articles # sent # words PEN 100 2.90 30.5 NATURE 100 2.79 29.9 GEO 100 2.53 28.4 RECV 100 2.76 27.0 Table 2: Statistics for the collected datasets. pilot studies also showed that annotators are often unsure where the cutoff is for information to be notable enough to include in a summary. We therefore view this disagreement as inherent to the task, and preserve these disagreements in evaluation rather than computing a consensus summary. We also compare the overlap between aspect-oriented annotation and generic extractive oracle derived from reference summaries from CNN/DM. In Table 4, the similarity and exact match 4 between generic oracle summaries and the top 3 annotated sentences are fairly low, which means the annotated aspect driven summaries significantly differ from the standard extractive oracle. Our aspect-oriented data collection works well to create labeled evaluation data, but it is difficult to scale to produce a large training set. Identifying suitable domains and specifying user intents requires significant human effort, and collecting real test cases at scale would require a more involved user study. We build an aspect-oriented model without gold-labeled aspect-oriented training data. We do this by generating keywords for each article in CNN/DM, and training the model to learn the relationship between these keywords and a summary. Our system follows broadly similar principles to He et al. (2020), but in an extractive setting. We present a scheme to generate keywords for each document from the original dataset. CNN/DM consists of pairs ( D, S ) of a document D and associated summary S . We aim to augment these to form ( D, K, S (cid:48) ) triples with keywords K and a possibly modified summary S (cid:48) . Our mixed augmentation technique requires training the model on both ( D, S ) and ( D, K, S (cid:48) ) for a given document. We now describe the steps to create this data. Keyword Extraction For each document in CNN/DM, we calculate the most important tokens in that document according to their TF-IDF ranking with respect to the entire corpus. Of these tokens, we select the ones that are present in the reference summary. This process selects tokens that are more likely to be consequential in affecting the output summary. we need to derive extractive oracle summaries for training; these consist of sentence-level binary decisions E = E 1 , . . . , E m for each sentence. Traditionally, this is done by finding a set of sentences that maximize ROUGE-2 (R2) with respect to the reference: argmax ER 2( E , S ) (Gillick and Favre, 2009; Nallapati et al., 2017). However, training the model to predict P ( S 1 , . . . , S m | D, k ) , an extractive analogue of He et al. (2020), was insufficient for our extractive model to learn to be sensitive to keywords; it merely learned to return a good generic summary regardless of what keywords were given. To instill stronger dependence on the keywords, we made two modifications to this process. First, we modified the reference summary by concatenating the keywords with the reference summary before computing the extractive oracle summary. This concatenation makes the oracle extraction more likely to select sentences containing the keywords, though modifying the reference summary requires maintaining a balance between the influence of keywords and of the original gold summary. Second, we use BERTScore (Zhang et al., 2020b, BS) rather than ROUGE-2 to identify sentences that closely match the reference summary. BERTScore turns out to boost the evaluation performance by a large margin, as shown in Table 12, so we use BERTScore for oracle extraction for all our experiments. One reason for this is that the ROUGE-2 summaries favor exact keyword matches in selecting sentences, so the trained model simply learned to keyword matching in extreme cases. Our final reference summary is therefore argmax EBS ( E , S + nK ) , where n is a hyper-parameter we discuss next. Keyword Intensity To compute n , we introduce another parameter r that controls the ratio of keyword tokens to original reference summary tokens. Higher values of r lead to extracting sentences in a manner more closely approximating keyword matching, but yielding poor standalone summaries. On the other hand, lower values of r may lead to generic summaries insensitive to the keywords. In practice, the number of times a keyword w is concatenated to the original summary S is defined as n = r  len ( S ) #( keywords ) where len ( S ) is the number of tokens in the original summaries and #( keywords ) is the total number of keywords available. When r = 1 , the concatenated keywords have the same length of the original summary. Mixed Training We explore a variant of training where we include training data with multiple variants of each original document from the dataset. Each document in the original dataset is mapped to two training samples, (1) a document without keywords and an unmodified oracle extractive summary, (2) a document with keywords and an oracle extractive summary using our modification procedure. Our model is trained to predict a summary S from a document-keywords pair ( D, K ) . Following BERTSUM (Liu and Lapata, 2019), we fine-tune BERT (Devlin et al., 2019) for extractive summarization using our modified CNN/Daily Mail dataset with keywords. During training, we prepend a special token followed by the keywords to the original document, and use the modified oracle extractive summary as the gold outputs. During inference, the keywords are user-defined. This scheme is similar to He et al. (2020), but differs in that it is extractive. We refer to this model, trained on our BERTScore references with the mixed training scheme, as AOSUMM . We evaluate our model on the ASPECTNEWS dataset, comparing performance on aspect-oriented summarization to several baselines. We additionally experiment on the SPACE multi-document dataset (Angelidis et al., 2021) to provide a point of comparison on a prior dataset and show that our aspect-oriented method is competitive with other systems. On ASPECTNEWS , we evaluate our model against the annotations using using F 1 score and ROUGE scores. It is impossible to achieve 100 F 1 on this task due to inherent disagreement between annotators. One downside of F 1 is that the model may be penalized even when the predicted sentence is very similar to the annotation, for this reason we also calculate ROUGE-1, -2, and -L scores (Lin, 2004). On the SPACE dataset, the gold summaries are abstractive, so we only calculate ROUGE scores. On the SPACE corpus, we primarily focus on comparisons to quantized transformer (QT) (Angelidis et al., 2021) and CTRLSUM (He et al., 2020). For the ASPECTNEWS dataset, we benchmark our system against several other models and baselines which we now describe. Heuristic and QA Baselines KEYWORD takes the keywords described in Table 1 and greedily finds the first occurrence of each keyword in the input document. STDREF stands for the extractive oracle given the original reference summaries from CNN/DM. QA uses an ELMo-BiDAF question answering model (Seo et al., 2017; Peters et al., 2018) to find answers to synthetic questions  What is {keyword}?  for each keyword in the article. We select the sentence where the selected span is located as a sentence to extract. Each of these three technique is an extractive baseline where top sentences are selected. Summarization Baselines We also compare our AOSUMM model against text summarization models, and query-focused models from previous work (retrained or off-the-shelf). (i) BERTSUM is a bert-base-cased extractive summarization model fine-tuned on CNN/DM (Liu and Lapata, 2019). (ii) BERT-FK shares the similar model architecture as BERTSUM but the training data comes from Frermann and Klementiev (2019). This data is constructed by interleaving several articles from the CNN/DM dataset together, extracting a coarse aspect from the original URL of one of the article, and setting the new gold summary to match that article. (iii) CTRLSUM is an off-the-shelf abstractive summarization model with the capability of conditioning on certain queries or prompts (He et al., 2020). (iv) Our model AOSUMM is based on BERTSUM and trained with techniques described in Section 4. 5.3 Results ASPECTNEWS The experimental results on ASPECTNEWS are shown in Table 6. We find that our model outperforms our baselines across F 1 , ROUGE-1, ROUGE-2, and ROUGE-L scores. Significantly, our model generally outperforms keyword matching, demonstrating that semantic match information from training with the BERTScore oracle may be more useful than training with a ROUGE oracle in terms of reproducing annota-tors' judgments; recall that our model has not been trained on any ASPECTNEWS data and only on our synthetic data. We note that our model's performance falls behind keyword matching some baselines in the geography aspect; this may be because the aspect is relatively homogeneous and can be easily approximated by keyword matching. SPACE The results on all the aspects of the SPACE dataset are shown in Table 7. All of the aspect-oriented models exceed the performance of the generic summaries produced by BERTSUM . We also find that our model performs competitively with the quantized transformer (QT) (An-gelidis et al., 2021) and CTRLSUM (He et al., 2020) methods in this dataset. This is a surprising result: the AOSUMM model is trained only with out-of-domain synthetic data, without access to the aspects prior to keywords specified at test time. Additionally, this is an abstractive task that we are applying an extractive model to. Keyword Sensitivity We evaluate the sensitivity of the model to different keywords. There is KWF 1 R-1 R-2 R-L F 1 R-1 R-2 R-L PENANNOTNATUREANNOTPEN 44.8 64.2 54.1 51.6 41.8 60.8 49.5 46.5 NATURE 44.3 65.5 56.0 51.3 45.2 64.4 53.9 48.0 GEOANNOTRECVANNOTGEO 49.9 69.1 61.2 54.2 38.0 56.2 45.3 46.2 RECV 42.8 60.4 49.7 47.8 39.6 59.5 49.1 46.7 Table 8: Keyword sensitivity analysis broken down by domain of ASPECTNEWS . some overlap between the summaries returned by different keyword sets, as shown by the Jaccard similarity: some sentences may fit under both GEO and RECV , or both PEN and NATURE . Table 9 shows statistics of this, with the Fraud keyword sets yielding more similar summaries than those in Earthquake. We also confirm that using the keywords matched to our setting outperforms using other sets of keywords in that domain (Table 8) suggesting that our model is picking summaries in a keyword-driven fashion. Keyword Intensity We can vary the parameter k controlling the number of times we append the keywords to the reference summary in order to generate the oracle extractive summary. We experiment with different level of intensity and show the result in Table 10. For most cases, r = 1 works well among all the datasets. Extractive vs. Abstractive Comparison It is difficult to directly compare the quality of summaries produced by an extractive model to those produced by an abstractive model. Abstractive models do not extract individual sentences from a summary so direct F 1 evaluations cannot be compared in the manner of Table 6. ROUGE scores are a misleading comparison given that an extractive model will be better matched to our extractive ground truths. Therefore, we perform a qualitative analysis to determine the models' relative responsiveness to keywords and relative advantages and disadvantages. 5 Keyword Sensitivity Comparison Although both CTRLSUM and AOSUMM are sensitive to the choice of keywords and alter their summary in response to different keywords, CTRLSUM often either hallucinates false information (Maynez et al., 2020) or simply rewords the prompt in the generated summary. We found that just under the GEO keywords in the earthquakes domain, out of 100 sample articles the bigram not known appears 27 times in relation to describing the location of the earthquake and not immediately known appears another 24 times. The CTRLSUM model frequently rephrases the prompt rather than synthesizing information in the document related to the keywords into a cogent summary. Comparison of Factuality of Output Table 11 shows one example of CTRLSUM hallucination in the GEO case. Here, the model also rewords the prompt and inserts it into the summary without 5 Note that for the abstractive SPACE dataset we considered here, we found that the performance difference between our model and abstractive models is small. Our investigation found that, at least on this dataset, abstractive models are engaging in heavy copying of the source text, suggesting that extractive models may be almost as well suited for this task as abstractive models. adding new information. Although such behavior may possibly perform well on automated metrics, it does not serve the purpose of query-focused summarization. Extractive summaries Table 11 shows that our model is able to successfully extract relevant parts of the document for our aspects under consideration. There are some features which may make these summaries hard to process in isolation, such as the quake in the first R sentence; our method could be extended with prior techniques to account for anaphora resolution (Durrett et al., 2016). In this paper, we present a new dataset for aspect-oriented summarization of news articles called ASPECTNEWS . Unlike query-focused summarization datasets which are often driven by document specific facts or knowledge, this aspect-oriented task is designed to mimic common user intents in domain-specific settings. We present a keyword-controllable system trained on synthetic data and show that it can perform well on ASPECTNEWS without training on the target domains, performing 6501 better than a range of strong baseline methods. This work was chiefly supported by funding from Walmart Labs and partially supported by NSF Grant IIS-1814522, a gift from Amazon, and a gift from Salesforce Inc. Opinions expressed in this paper do not necessarily reflect the views of these sponsors. Thanks to Ido Dagan for helpful discussion and suggestions about this paper, as well to the anonymous reviewers for their thoughtful comments.
abstain abstain method method abstain result objective result abstain abstain other abstain other abstain abstain objective method abstain abstain method objective objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result result other other other abstain abstain other other abstain other abstain other other objective method other other other other other other other method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method abstain method abstain method method method method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method method method method abstain abstain method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result other other other