The availability of large-scale datasets has driven the development of neural models that create generic summaries from single or multiple documents. In this work we consider query focused summarization (QFS), a task for which training data in the form of queries, documents, and summaries is not readily available. We propose to decompose QFS into (1) query modeling (i.e., finding supportive evidence within a set of documents for a query) and (2) conditional language modeling (i.e., summary generation). We introduce MARGE , a Ma sked ROUGE Regression framework for evidence estimation and ranking which relies on a unified representation for summaries and queries, so that summaries in generic data can be converted into proxy queries for learning a query model. Experiments across QFS benchmarks and query types show that our model achieves state-of-the-art performance despite learning from weak supervision. 1 1 Introduction The neural encoder-decoder framework has become increasingly popular in generic summarization (See et al. 2017; Gehrmann et al. 2018; Liu and Lapata 2019a; Fabbri et al. 2019, inter alia ) thanks to the availability of large-scale datasets containing hundreds of thousands of document-summary pairs. Training data of this magnitude is not readily available for query focused summarization (QFS; Dang 2005) which aims to create a short summary from a set of documents that answers a specific query. Existing corpora (Nema et al., 2017; Dang, 2005; Hoa, 2006; Baumel et al., 2016) are relatively small for modern data-hungry neural architectures and have been mostly used for evaluation purposes. A major bottleneck in leveraging generic summarization data for QFS is the absence of queries (Nema et al., 2017); the majority of existing datasets consist of document-summary pairs, while QFS summaries are expected to answer specific queries. Recent work (Xu and Lapata, 2020; Su et al., 2020; Laskar et al., 2020) sidesteps this problem by resorting to distant supervision from query-relevant NLP resources including question answering (Rajpurkar et al., 2016; Chakraborty et al., 2020) and paraphrase identification (Dolan and Brockett, 2005). Such approaches incorporate query modeling in the summarization process but are even more data hungry compared to generic summarization ones, since they additionally require access to QA datasets which can be extremely costly to create (Bajaj et al., 2016; Kwiatkowski et al., 2019). Moreover, there is often a mismatch between queries in QA datasets and those in QFS scenarios (Xu and Lapata, 2020); the two types of queries are not identically distributed and it is practically infeasible to find appropriate query-related resources for all domains and topics. In this work we do not assume access to any resources other than those available for generic summarization. We further decompose abstractive QFS into two subtasks: (1) query modeling (i.e., finding supportive evidence within a set of documents for a query) and (2) conditional language modeling (i.e., generating an abstractive summary based on found evidence). Under this formulation, we use generic summarization data not only for conditional language modeling, but also for learning an evidence ranking model. Inspired by the Cloze task and its applications in NLP (Taylor, 1953; Lewis et al., 2019; Lee et al., 2019), we propose MARGE , a Ma sked R OU GE regression framework for evidence estimation and ranking. MARGE intro-Masked Summary The Da Vinci Code was published in 2003, and within six years Brown had booted John Grisham from the No. 1 slot on the list of writers whose books were most often donated to Oxfam's 700 shops. duces a unified representation for summaries and queries , so that summaries in generic data can be converted into proxy queries for learning a query model. Based on the evidence selected by MARGE , we generate abstractive summaries whilst controlling their length and the extent to which the query influences their content. Our contributions in this work are threefold: we propose a weakly supervised system for abstractive QFS where no query-related resources are required; we discover a new type of connection between generic summaries and QFS queries, and provide a universal representation for them which allows generic summarization data to be exploited for QFS; we provide experimental results on QFS benchmarks, and show that across query types and domains our system achieves state-of-the-art results on both evidence ranking and abstractive QFS. The majority of previous QFS approaches have been extractive, operating over queries and document clusters from which they select query-relevant sentences to compose a summary. They mostly differ in the way centrality and relevance are estimated and incorporated, e.g., via manifold ranking (Wan et al., 2007), using a look-ahead strategy (Badri-nath et al., 2011), uncertainty prediction (Wan and Zhang, 2014), or attention mechanisms (Li et al., 2017a,b). More recently Xu and Lapata (2020) propose a coarse-to-fine framework that leverages distant supervision from question answering to extract summary-worthy content. Abstractive QFS has received significantly less attention. This is due to generation models being particularly data-hungry (Lebanoff et al., 2018; Liu and Lapata, 2019a) and the scarcity of QFS training data. The increasing availability of pretrained models has prompted the development of pipeline-style frameworks for QFS which use resources from a wider range of NLP tasks. For example, Su et al. (2020) fine-tune BART (Lewis et al., 2020) on CNN/DailyMail (Hermann et al., 2015), a single-document summarization dataset, and generate abstracts for QFS by iteratively summarizing paragraphs to a budget. They learn a query model for paragraph selection based on a plethora of QA and machine reading datasets (Su et al., 2019; Ra-jpurkar et al., 2016). Similarly, Laskar et al. (2020) fine-tune BERTSUM on CNN/DailyMail, and propose a three-stage system which uses supervision from QFS data (typically reserved for evaluation) and related QA and paraphrase identification tasks. We also focus on abstractive QFS, however, we do not assume access to any additional training resources over and above generic summarization datasets, even for query modeling. Moreover, our system is able to generate long QFS abstracts all at once , instead of iteratively creating bullet-style summaries which often lack coherence. Let { ( S, D ) } denote a generic summarization dataset where D = { d 1 , d 2 , . . . , d M } is a collection of documents with corresponding summaries S . |D| = 1 for single-document summarization (SDS) and |D| > 1 for multi-document summarization (MDS). In QFS, a query Q additionally specifies an information request, { ( S, D , Q ) } . It is often assumed (e.g., in DUC benchmarks) that Q consists of a short title (e.g., Amnesty International ), and a query narrative which is longer and more detailed (e.g., What is the scope of operations of Amnesty International and what are the international reactions to its activities? ). In this work, we propose to decompose QFS into two sub-tasks, namely query modeling and conditional language modeling . The query model q  ( D | Q ;  ) estimates whether textual units (e.g., sentences) within document cluster D are relevant to query Q , while p  ( S | D, Q ;  ) generates summary S conditioned on evidence provided by the query model and (optionally) the query itself (see Figure 1(b) for an illustration). When S  Q , we have a query-agnostic conditional language model p  ( S | D ;  ) . Otherwise, the conditional language model is query-guided . Our query model is trained with distant supervision derived from generic summarization data which is easier to obtain (e.g., from online sources) compared to QA datasets which must be annotated from scratch (e.g., for different types of questions and domains). Although queries are not verbalized in generic summarization, we hypothesize that the summaries themselves constitute a response to latent queries. So, how can we reverse-engineer the queries from the summaries? Inspired by the standard Cloze task (Taylor, 1953) and its recent variants (Lewis et al., 2019; Lee et al., 2019), we render queries and summaries in a Unified Masked Representation (UMR) which enables summaries to serve as proxy queries for model training, as shown in Figure 1(a). We further assume that the answer to these queries can be found in sentences which form part of the document collection D . Although we do not know for certain what these sentences are we can assume that if they have a high ROUGE score against the reference summary they are likely to contain an answer. We therefore use ROUGE as a distant supervision signal, and train a model that takes a query and document sentence as input and estimates their relevance. At inference time, we also render actual queries in UMR and rank all sentences in the document collection with our trained model. The most relevant sentences serve as input to a conditional language model to generate query focused abstractive summaries. As explained earlier, we train a query model q  ( D | Q ;  ) on summary-sentence pairs via distant supervision. We use a summary-based proxy query UMRS during training and an actual query UMRQ during testing. In the following, we first describe how UMRs are obtained and then discuss how the query model is trained. Unified Masked Representation The intuition behind UMR is that a summary will encapsulate most salient information a user needs, while a query typically covers only a small fraction. We thus add one or more placeholders to the query to represent missing information the user actually seeks. We also identify such information in generic summaries for selective masking, to reduce the distributional shift during training. The UMR for a summary is the concatenation of its sentential UMRs. To convert a sentence from natural language to UMR, we parse it with Open Information Extraction (Open IE; Stanovsky et al. 2018) to a set of propositions consisting of verbs and their arguments. The latter are considered candidate information slots I . We initialize Algorithm 1, by replacing all such slots with a [MASK] token. We subsequently sample and reveal a set of slots subject to a budget constraint. We define the budget as B =   |I| where   [0 , 1] modulates the proportion of tokens to be revealed within I slots (and is optimized on the development set). Finally, in order to keep the representation of UMRS and UMRQ consistent (see next para-graph), we merge adjacent [MASK] tokens to one [MASK] resulting in a partially masked summary. We mask QFS queries by considering their structure and lexical makeup. Queries in DUC benchmarks often contain interrogative words (e.g., how is A and what is B ) and request words (e.g., describe A and tell me B ). Following this observation, we manually collect a small set of such query words and replace them with [MASK] . For queries with a title and a narrative, we first mask the narrative and then prepend  [MASK] T . , where T is a sequence of title tokens. Figure 1(a) shows examples of a masked query and summary. Evidence Ranking We represent sentences in a document collection and UMR queries with a pretrained BERT model (Devlin et al., 2019). Specifi-cally, we concatenate a UMR query and a candidate sentence to sequence  [CLS] U [SEP] C [SEP]  where U is a sequence of tokens within a UMR query and C a sequence of tokens in a document sentence (we pad each sequence in a minibatch of L tokens). The [CLS] vector serves as input to a single layer neural network which estimates whether the sentence contains sufficient evidence to answer the query (see Figure 1(b) right). We use the mean-square error to compute the loss and update the encoding parameters in BERT via standard backpropagation: L (  ) = 1 |D| (cid:88) ( S,C ) D (cid:104) ( y   y ( S, C ;  )) 2 (cid:105) . where S, C is a summary-sentence pair sampled from collection D and y the training signal. Recall the summary is rendered as UMRS . Previous work (Liu and Lapata, 2019a) has used ROUGE-2 as training signal for paragraph ranking. However, sentences are significantly shorter than paragraphs, and we observe a number of instances with a ROUGE-2 score of 0. We therefore perform label smoothing and define y as the F1 interpolation of ROUGE-2 and ROUGE-1: y = R 2 ( S, C ) +   R 1 ( S, C ) where  is optimized on the development set. At inference time, we use the trained model to compute the affinity score between UMRQ and all candidate sentences in D and rank them accordingly. The highest ranked sentences are deemed query-relevant and passed on to our summary generation model. 2 Query Narrative Expansion In some cases queries may be relatively short and narratives absent. This can be problematic for our setup since query proxies (in the form of summaries) are typically long and detailed. For datasets with short queries we automatically create query narratives in an unsupervised fashion. We employ LexRank (Erkan and Radev, 2004) to select a subset of representative sentences under a word budget and concatenate them to form narratives (which we append to the original queries). 2 The Cloze task has been also employed in recent work in generic summarization (Huang et al., 2020). In comparison, we address a different research question (i.e., query modeling vs. summary evaluation) based on a different formulation (masked ROUGE regression vs. multiple-choice QA). We also leverage generic summarization datasets to fine-tune a pretrained language model for abstractive QFS. In experiments we employ the publicly released UNILMV 2 (Bao et al., 2020) to instantiate the controllable generator shown in Figure 1(b), however any other language model could have been used instead. With Transformer (Vaswani et al., 2017) as the backbone network, UNILMV 2 is jointly pretrained for natural language understanding and generation. Specifically, a bidirectional model is employs an autoencoding objective (AE; identical to Devlin et al. 2019), while a partially autoregressive (PAR) sequence-to-sequence model decomposes the probability of masked tokens in input sequence x as: p ( x M | x \ M ) = | M | (cid:89) i =1 (cid:89) m  M i p ( x m | x \ M  i ) (2) where M is the uniformly-produced factorization order. The masked position set M i at the i th factorization step can be either a token or a n -gram block. x M is a set of x M i , and similarly, x \ M is a set of x \ M i . The pretraining loss is computed as LAE + LPAR . At inference, UNILMV 2 operates over sentences deemed relevant by the query model and decodes summaries autoregressively (see Figure 1(b) left). Synthetic MDS Data The pre-trained language model can be fine-tuned on MDS datasets (e.g., Multi-News; Fabbri et al. 2019) which are perhaps better aligned with the QFS task since both MDS and QFS operate over document clusters. We additionally propose a way to create synthetic MDS datasets based on SDS data. This is advantageous for two reasons. Firstly, MDS resources are fairly limited compared to SDS data (Zhang et al., 2018; Lebanoff et al., 2018). And secondly, by construction, we can ensure various data characteristics which might be desirable (e.g., the number of topics represented in the document collection). A challenge with leveraging SDS for QFS is the summary length (Lebanoff et al., 2018). Summaries in SDS datasets such as CNN/DailyMail (Hermann et al., 2015), are on average 30 tokens long. In contrast, query focused summaries can be as long as 250 tokens. We sidestep this problem by adopting a retrieval -based solution. Specifically, we first build a database with all summaries in the original dataset. For each sample ( d i , s i ) , we query the database with summary s i . We retrieve N i  1 other summaries S i with the bigram hashing and TF-IDF matching method described in Chen et al. (2017). Then, we fetch their corresponding articles D i , and form the i th cluster as: D  i = { d i } (cid:91) D i (3)  s  i = concat( s i , , s i, 1 , . . . , s i,N i ) , s i,n  S i (4) where D  i are the source documents, and  s  i is a potentially redundant summary of them. We set N i to minimize the length difference between  s  i and our summary length requirement (e.g., 250 tokens). To obtain the final summary s  i , we eliminate redundancy by selecting sentences from the start of  s  i , skipping sentences that have high cosine similarity with those which have already been selected. Summarization Input In generic MDS, the input to the summarization model is a long sequence, i.e., documents within a cluster are concatenated together and sentences in each document follow their original order (Fabbri et al., 2019). In QFS, information about absolute (document) position is lost after evidence ranking. As a result, there is a discrepancy between training and testing for our generation model. To mitigate this, we collect all sentences across documents for each training sample and rank them in descending order according to their ROUGE-2 score against the reference summary. The pretrained language model is fine-tuned against this evidence-ranked list of sentences. During inference, when actual queries are available, we instead use the top sentences ranked by our query model as input to summary generation. Query Guidance Given that summarization input essentially consists of sentences that are highly relevant to the query, an obvious question concerns the usefulness of explicitly modeling the query during generation. We thus instantiate two conditional language models. For a query-guided summarizer p  ( S | D, Q ;  ) , we prepend UMRSS to the selected evidence during training and UMRQ at inference. While for a query-agnostic summarizer p  ( S | D ;  ) , we only consider the selected evidence as input to our summarizer and this setting is identical to generic MDS. Length Control QFS tasks usually require summaries of a fixed length budget (e.g, 250 words), whereas summary length is bound to be variable Dataset 2005 2006 2007 TD-QFS Domain Cross Cross Cross Medical Query Narrative Long Long Long Short #Clusters 50 50 45 4 #Queries/Cluster 1 1 1 10 #Documents/Cluster 32 25 25 185 #Summaries/Query 4-9 4 4 3 #Words/Summary 250 250 250 250 Table 1: Multi-Document QFS dataset statistics. in the training data. Inspired by Fan et al. (2018), we quantize summary length into discrete bins. We augment each training instance with this information, i.e., we prepend a length token (e.g., [230] ) to document sentences. At inference, we inform the model of the summary budget by prepending the expected length token (e.g., [250] ) to the sentences selected by the evidence ranker (see Figure 1(b)). Datasets We performed experiments on the DUC 2005-2007 QFS benchmarks and TD-QFS (Baumel et al., 2016). DUC benchmarks contain long query narratives while TD-QFS focuses on medical texts with short keyword queries. Statistics for both datasets are given in Table 1. We used DUC 2005 as a development set to optimize hyperparameters and select abstractive models, and evaluated performance on the other three datasets. We used Multi-News (Fabbri et al., 2019) and CNN/DailyMail (Hermann et al., 2015) as our generic summarization datasets to train MARGE (for evidence ranking) and to fine-tune UNILMV 2 (for summary generation). Data statistics are shown in Table 2. To create the training and development sets for optimizing MARGE , we sampled sentences from each dataset. Specifically, we took the first and last 20 sentences from each cluster in Multi-News and the first and last three sentences from each article in CNN/DailyMail. For fine-tuning UNILMV 2, we used the original Multi-News and the synthetic multi-document version of CNN/DailyMail described in Section 5. Implementation Details We used the publicly released BERT model 3 and fine-tuned it for ROUGE regression with a learning rate of 3  10  5 and a batch size of 128 for 3 epochs on 8 GPUs (GTX 2080 Ti). We trained two summarization 3 https://github.com/huggingface/pytorch-transformers Query Modeling Multi-News CNN/DM #Sentence/Doc 20 3 #Train 1,615,508 1,719,210 #Validation 200,824 80,052 #Words/Proxy Query 111.7 26.0 #Masks/Proxy Query 35.6 8.1 Summary Generation Multi-News CNN/DM #Clusters 44,972 287,227 #Documents/Cluster 2.8 4.1 #Words/Summary 257.2 261.3 Table 2: Training data for query modeling and summary generation. models on CNN/DailyMail and Multi-News, respectively, with the same hardware. For both models, we set the maximum input length to 768, and fine-tuned the publicly released UNILMV 2 model 4 with a learning rate of 7  10  5 and a batch size of 16 for 40,000 steps with gradient accumulation every 4 steps. During decoding, we used beam search with beam size 5 and Trigram Blocking (Paulus et al., 2018) to reduce redundancy. The cosine similarity threshold for redundancy removal was set to 0.6 and summary length was discretized to 10 bins. The  parameter for label smoothing was set to 0.15. We set  , the parameter which modulates the proportion of information slots to reveal during masking, to 0 (see Appendix for detailed analysis of  and its effect on model performance). Our experiments evaluate both components of the proposed approach, namely query modeling and summary generation. We assess the evidence ranker and the effectiveness of the unified masking. We also compare our summaries against competitive abstractive and extractive systems using automatic and human-based evaluation. Evaluation Metrics We evaluate query modeling with retrieval and summarization metrics. For the former evaluation, we follow Liu and Lapata (2019a), concatenate the top k ranked sentences, and calculate recall against gold summaries. We additionally propose to evaluate model output as 4 https://github.com/microsoft/unilm Models DUC 2006 DUC 2007 TD-QFS R@10 R@30 R@10 R@30 R@10 R@30 ORACLE 6.7 16.2 8.4 19.1 17.2 35.6 TERMFREQ 7.2 15.1 8.5 18.5 14.2 25.9 BERTQA 8.5 16.3 10.2 20.2 9.8 21.9 BERTMRC 8.2 16.6 9.0 19.2 8.1 16.4 MARGE-MN 11.1 20.2 13.8 25.3 11.2 21.6 + EXPAND     18.1 32.9 MARGE-CD 9.1 17.4 11.1 22.1 10.0 18.7 + EXPAND     17.2 27.7 Table 3: Retrieval performance of evidence rankers. if it were an extractive summary, to better assess coverage and informativeness. We thus take the top sentences subject to a budget of 250 tokens, and remove redundancy by selecting sentences from the top and skipping sentences that have high cosine similarity (e.g.,  0 . 6 ) with selected ones. We use ROUGE F1 to evaluate the resulting summaries so that precision is also taken into account. Results We compare MARGE against Term Frequency, a simple but effective retrieval method that performs particularly well on DUC datasets (Katra-gadda and Varma, 2009). We also compare to two semantic matching models used for extractive QFS (Xu and Lapata, 2020): BERTQA which is trained on the joint set of WikiQA (Yang et al., 2015) and TrecQA (Yao et al., 2013), and BERTMRC which is fine-tuned on SQuAD 2.0 (Rajpurkar et al., 2018). ORACLE uses reference summaries as queries to retrieve summary sentences. For summarization evaluation, we report upper bound performance (GOLD ) which we estimated by comparing a (ran-domly selected) reference summary against the remaining three reference summaries. In addition, we compare to LEAD which returns all lead sentences of the most recent document (up to 250 words) and LEXRANK (Erkan and Radev, 2004), a widely-used unsupervised method based on Markov random walks on sentence-similarity graphs. 5 We summarize ranking and summarization results in Tables 3 and 4. As we can see, despite learning from weak signals, i.e., proxy queries and proxy answers, MARGE outperforms the strongest base-5 To examine ranking performance, we exclude multi-stage frameworks like Xu and Lapata (2020) that rerank the evidence with additional modules (e.g., centrality). line, BERTQA , under both evaluation tasks. Without recourse to any question/answer annotations or dataset-specific retrieval methods, our model provides more informative input to the downstream generation task. As anticipated, query expansion ( + EXPAND ) gives a big boost on TD-QFS (which has short queries) leading to better coverage. Ablation Studies Table 5 shows the outcome of various ablation studies which assess the effectiveness of masking and how to best instantiate it. Specifically,  Verb additionally treats verbs as information slots for sampling and masking;  Mask removes masking entirely so that the whole summary is revealed;  Query removes the proxy query (at training time) and the actual query (at inference time); this is to investigate whether our model simply learns to judge sentence salience based on its own features, instead of performing semantic matching with the given query;  OpenIE removes the dependency on Open IE and chooses words to mask at random. Specifically, we randomly mask 15% words in summaries as in BERT (Devlin et al., 2019) and merge adjacent [MASK] tokens. Performance drops in all cases, especially when queries Models DUC 2006 DUC 2007 TD-QFS PQSUM-WSL  16.5 17.7  QUERYSUM  15.3 16.8 20.7 BART-CAQ 12.9 14.4  PQSUM 14.8 16.0  UNILM-MN 11.8 12.3 12.9 UNILM-CD 13.6 14.9 16.7 MARGESUM-MN 14.3 16.5 16.5 MARGESUM-CD 15.1 16.9 20.9 Table 6: Abstractive summarization models with R-SU4 (full set of results in Appendix);  /  : extrac-tive/supervised method. are removed, underscoring the effectiveness of the proposed representation and training framework. Automatic Evaluation Table 6 compares our model, which we call MARGESUM , against existing QFS systems. These include PQSUM-WSL (Laskar et al., 2020) a supervised abstractive system which represents the state of the art on DUC benchmarks. It first extracts relevant sentences for each document with a QA model, it then replaces some of these with reference summary sentences via a paraphrase model, and uses them to further fine-tune BERTSUM (Liu and Lapata, 2019b). In its supervised incarnation, two years' DUC datasets are used for training and one for testing. QUERYSUM (Xu and Lapata, 2020) is state-of-the-art extractive system which adopts a coarse-to-fine process for salience estimation. The second block compares our model with two distantly supervised approaches. BART-CAQ (Su et al., 2020) uses an ensembled QA model to extract answer evidence, and fine-tuned BART (Lewis et al., 2020) to iteratively generate summaries from paragraphs. PQSUM (Laskar et al., 2020), uses fine-tuned BERTSUM to generate summaries for each document in a cluster, and a QA model to rank summary sentences against the query. Table 7 compares these models and our own in terms of their training requirements. The third block presents the performance of UNILM fine-tuned on Multi-News and CNN/DailyMail following the standard setting in Bao et al. (2020). It uses no query guidance or length control. Documents are concatenated as input for training. During testing, sentences are selected with MARGE but ordered according to Models QA PI GS QFS BART-CAQ (Su et al., 2020) (cid:51) (cid:55) (cid:51) (cid:55) PQSUM (Laskar et al., 2020) (cid:51) (cid:55) (cid:51) (cid:55) PQSUM-WSL (Laskar et al., 2020) (cid:51) (cid:51) (cid:51) (cid:51) UNILM (Bao et al., 2020) (cid:55) (cid:55) (cid:51) (cid:55) MARGESUM (cid:55) (cid:55) (cid:51) (cid:55) Table 7: Training requirements for existing QFS models (QA, PI, GS, and QFS stand for question answering, paraphrase identification, generic summarization and query focused summarization). their original document position. The last block shows two variants of MARGESUM , optimized on Multi-News and a synthetic training set built from CNN/DailyMail. Both take as input sentences selected with MARGE-MN during inference. As we can see, without requiring expensive QA data (see Table 7), MARGESUM-CD outperforms existing distantly supervised approaches. Its performance on DUC is on par with one of the strongest extractive systems, while on TD-QFS it is supe-rior across metrics. Also note that MARGE trained on synthetic MDS data outperforms MARGESUMMN . Compared to Multi-News, synthetic summaries cover more topics and are less redundant, which is suited to QFS where there are usually multiple sub-queries to answer. Ablation Studies Table 8 presents the results of several ablation studies on MARGESUM-CD . Replacing the input to the summarization component with sentences selected by BERTQA (Xu and Lapata, 2020) significantly decreases performance, demonstrating that sentences selected by MARGE are useful to downstream abstractive summarization. Removing evidence ranking altogether (  Rank) leads to a large performance drop; this is expected since sentence position information from the original documents does not transfer well to QFS settings. Removing length control (  Length) also hurts performance as does the removal of query guidance (  Query) at inference time. Human Evaluation We also evaluated model summaries in a judgment elicitation study via Amazon Mechanical Turk. Native English speakers (self-reported) were asked to rate query-summary pairs on two dimensions: Succinctness (does the summary avoid unnecessary detail and redundant information?) and Coherence (does the summary make logical sense?). The ratings were obtained using a fivepoint Likert scale. In addition, participants were asked to assess the Relevance of the summary to the query. Crowdworkers read a summary and for each sentence decided whether it is relevant (i.e., it provides an answer to the query), irrelevant (i.e., it does not answer the query), and partially relevant (i.e., it is not clear it directly answers the query). Relevant sentences were awarded a score of 5, partially relevant ones a score of 2.5, and 0 otherwise. Sentence scores were averaged to obtain a relevance score for the whole summary. Participants assessed summaries created by PQSUM-WSL , the state-of-the-art abstractive system, QUERYSUM , a state-of-the-art extractive system, UNILM-CD , and MARGESUM-CD . 6 We also randomly selected GOLD standard summaries to include as an upper bound. We sampled 20 query-cluster pairs from DUC (2006, 2007; 10 from each set), and 20 pairs from TD-QFS (5 from each cluster) and collected three responses per pair. 6 We are grateful to Md Tahmid Rahman Laskar for providing us with the output of their PQSUM-WSL system. We include PQSUM-WSL only for human evaluation on DUC since it was not evaluated on TD-QFS (Laskar et al., 2020) and system output is not available. Table 9 shows the human ratings for each system (we provide examples of summary output in Appendix C). Participants perceive MARGESUMCD on par with PQSUM-WSL in terms of query relevance and summary succinctness, while significantly better than PQSUM-WSL and QUERYSUM in terms of coherence. In fact, participants find summaries PQSUM-WSL summaries as incoherent as those created by the extractive QUERYSUM ; this is probably due to the fact that PQSUMWSL first generates an abstractive summary for each document and then re-ranks the generated sentences. Therefore, final summary sentences are less related to each other. Summaries from our system are also considered significantly more relevant than UNILM-CD . Compared to PQSUM-WSL , although UNILM-CD is not good at producing relevant content, it maintains relatively higher coherence, demonstrating the effectiveness of training abstractive systems with synthetic data from SDS and generating long summaries at once. In this work we proposed an abstractive framework for query focused summarization. We provided a unified mask representation for summaries and queries, which enables summaries to serve as proxy queries for model training. As a result, a query model can be trained with generic summarization data without relying on additional question-answering resources. Experimental results across datasets show that the proposed system yields state-of-the-art performance despite the weakly supervised setting, and produces more relevant and coherent summaries compared to existing approaches. In the future, we would like to push this low-resource approach even further and attempt to generate abstractive summaries without access to any summarization datasets. The authors would like to thank the anonymous reviewers for their valuable feedback. We acknowledge the financial support of the European Research Council (Lapata; award number 681760). This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract FA8650-17-C-9118. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation therein.
abstain method objective method result abstain abstain abstain abstain abstain abstain abstain method result method abstain abstain abstain method other other other other other other other other other other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain method method abstain method abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method other abstain abstain abstain abstain abstain method abstain method method method method method method other abstain method method abstain abstain method abstain abstain abstain abstain abstain other abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain other abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method method result abstain abstain abstain method abstain objective method abstain abstain method other other other other other