Allowing users to interact with multi-document summarizers is a promising direction towards improving and customizing summary results. Different ideas for interactive summarization have been proposed in previous work but these solutions are highly divergent and incomparable. In this paper, we develop an end-to-end evaluation framework for interactive summarization, focusing on expansion-based interaction, which considers the accumulating information along a user session. Our framework includes a procedure of collecting real user sessions, as well as evaluation measures relying on summarization standards, but adapted to reflect interaction. All of our solutions and resources are available publicly as a benchmark, allowing comparison of future developments in interactive summarization, and spurring progress in its methodological evaluation. We demonstrate the use of our framework by evaluating and comparing baseline implementations that we developed for this purpose, which will serve as part of our benchmark. Our extensive experimentation and analysis motivate the proposed evaluation framework design and support its viability. Large bodies of texts on a topic oftentimes contain extensive information that is challenging for a potential reader to handle. Traditionally, information seeking tasks, like search, question-answering (QA) and multi-document summarization (MDS), are single-round input-output processes that can serve the information seeker only to a limited extent. This calls for an interactive setting where a user can guide the information gathering process. For search and QA, this type of research has been gaining momentum recently in areas such as exploratory search (Marchionini, 2006) and conversational QA (Reddy et al., 2019). For MDS, where interaction would allow a user to affect summary content, only sporadic works have been seen over the years (e.g., Leuski et al., 2003; Lin et al., 2010; Yan et al., 2011; Baumel et al., 2014; Christensen et al., 2014; Shapira et al., 2017; Handler and O'Connor, 2017). A key gap in the development and adoption of interactive summarization (denoted here INTSUMM ) solutions is the lack of evaluation methodologies and benchmarks for meaningful comparison of systems, similarly to those for static (non-interactive) summarization (e.g., NIST, 2014). The previous works on interactive or customizable summarization of multi-document sets are distinct, with proprietary evaluations that do not admit comparison. Furthermore, the evaluation processes are often not scalable and replicable, or do not give a comprehensive enough assessment. In this paper we develop an end-to-end evaluation framework for INTSUMM systems. The framework starts with real user session collection on a system, via a concrete process of controlled crowdsourcing that we designed for this task. The sessions are then measured to produce absolute scores for the system, allowing for robust system comparison. Our framework supports a general notion of expansion-based interactive summarization, where the textual summary gradually expands in response to user interaction. Figure 1 presents an INTSUMM system that we implemented to illustrate this notion (5.1). To ensure our evaluation framework is sound, we developed the framework in multiple cycles accompanied by user studies and extensive crowdsourcing experimentation. Our main contributions are as follows. (1) Evaluation measures. We propose a set of automatic and manual evaluation measures for INTSUMM systems, which build upon a combination of established notions in static summarization and interactive systems, and enable utilizing available multi-document summarization (MDS) Figure 1: Our INTSUMM web application, implemented for testing our evaluation framework. datasets. Our measures are aggregated over multiple interactive sessions and document sets to obtain an overall system evaluation. In contrast to static summarization, our measures apply to the steps along the interaction to reflect the progress of information acquirement rather than just its final result. This is done by converting an interactive session to a sequence of incrementally growing static summaries, and measuring the accumulating information gain with a recall metric. See 3. (2) Crowdsourced session collection process. Adequate INTSUMM system evaluation and comparison requires collecting realistic user sessions in a consistent manner, on which the measurements are conducted. Previous work mostly turn to in-house user-studies, which are less replicable, not scalable, and not always easily attainable. In contrast, standard crowdsourcing induces noise and overly tolerates subjective behavior, hindering replicability and comparability. We describe a controlled crowdsourcing procedure that overcomes the above obstacles, making the evaluation process reliable and much more accessible for researchers interested in pursuing INTSUMM research. See 4. We demonstrate the use of our full evaluation framework on two INTSUMM systems that we implemented, which apply different algorithms but share a common user interface, with the DUC 2006 (Dang, 2006) MDS dataset. Analysis shows favorable results in terms of internal consistency between sessions, users, and different evaluation measures, indicating that our solutions may serve as a promising benchmark for future INTSUMM research. See 5. The evaluation procedures and systems are available publicly. 1 2 Background Traditional MDS has been researched extensively (e.g. Goldstein et al., 2000b; Radev et al., 2004; Haghighi and Vanderwende, 2009; Yin and Pei, 2015). It encompasses variants of query-focused summarization (Dang, 2005), orienting the output summary around a given query (e.g. Daum III and Marcu , 2006; Zhao et al., 2009; Cao et al., 2016; Feigenblat et al., 2017; Baumel et al., 2018), and incremental update summarization (Dang and Owczarzak, 2008), generating a summary of a document set with the assumption of prior knowledge on an earlier set (e.g. Li et al., 2008; Wang and Li, 2010; McCreadie et al., 2014; Zopf et al., 2016). Evaluation approaches predominantly include automatic ROUGE (Lin, 2004) measurement, i.e. word overlap against reference summaries, and manual responsiveness (Dang, 2006) scores or pairwise comparison (Zopf, 2018) between summaries. In the related QA task (Voorhees et al., 1999), a system extracts an answer for a targeted question. Similarly, in the interactive setting, a conversational QA (Reddy et al., 2019) system extracts answers to a series of interconnected questions with a clear informational goal. To check correctness in both cases, a system answer is simply compared to the true answer via text-comparison. On the contrary, in the exploratory style of INTSUMM , where the knowledge desired is less certain, evaluation must consider dynamically accumulating information. Exploratory search (Marchionini, 2006; White and Roth, 2009) addresses the need for converting big data to knowledge via human-machine cooperation. For example, interactive information retrieval (Ingwersen, 1992) focuses on fine-tuning document retrieval interactively, and complex-interactive-QA (ciQA) (Kelly and Lin, 2007) involves interacting with a system to generate a passage that answers a complex question. Evaluation is a major challenge in dealing with these tasks (White et al., 2008; Palagi et al., 2017; Hendahewa and Shah, 2017). Firstly, real users must use the system being evaluated by completing a task-appropriate assignment, requiring large-scale user studies that highly increase the cost and complexity of evaluation. Furthermore, varying user behavior could mean distorted session comparison. Then, a system is measured on the basis of its final outputs, mostly disregarding the evolvement of the interactional session. Among interactive summarization, in the query-chain focused summarization task (Baumel et al., 2014), a chain of queries yields a sequence of short summaries, each refraining from repeating content. The task's evaluation relies solely on pre-defined sequences of queries with a respective reference summary per iteration (laboriously prepared by experts) that disregards previous outputs by the system. Other interactive summarization systems, such as Christensen et al. (2014); Shapira et al. (2017), present a preassembled summary with several levels of detail, allowing a user to drill down to or expand on information of interest. These works do not evaluate in a manner that is comparable to others, and do not consider information variation due to interaction. They perform small-scale user-studies for preference between their system and static variants, or a single automatic assessment of the fully expanded final summary. evaluation issues, specifically targeting the INTSUMM task, where the interaction-induced outputs are purely textual summary snippets of the input document set. An INTSUMM system is evaluated by measuring its performance on multiple sessions produced as a result of human operation. The input of a session,  , is a set of documents, D , on which to explore. A session comprises an automatically generated initial summary,  0 , and a sequence of user-posed requests, q i , and corresponding output responses, r i . The responses can be viewed as expansions of  0 . Consequently, the overall interactive summary resulting from  defines a sequence of incrementally expanding snapshots [  0 ,  1 , . . . ,  |  | ] where  i =  0  S ij =1 r j is the union of accumulative (summarized) information presented to the user after i interactions. Each snapshot may thus be regarded as a static summary, allowing static summarization measures to be applied on it. For compared INTSUMM systems S 1 , ..., S m , we require at least u sessions of distinct users interacting with S i on each test document set D  { D 1 , ..., D n } . Assuming such sessions, we next define automatic and manual evaluation measures, and defer details on adequate session collection to 4. Importantly, all measures are based on established evaluation mechanisms used in static summarization and interactive systems, that we extend or adapt for the INTSUMM setting, and that are practically linear in time to the length of the session sequence. Together, the set of measures we define provide an encompassing assessment adequate for the evaluation of interactive summary systems. Viewing a session as a sequence of incrementally expanding static summary snapshots, we would first like to obtain comparable scores for each static summary that will capture the information gained along the session up to the current interaction. Existing static MDS benchmarks provide reference summaries at a single length for the purpose of evaluating a summary at a similar length. This presumably means we would require a series of reference summaries that differ by small length gaps for the sequence of lengthening snapshots, which is diffi-cult and costly to produce. To address this obstacle, we leverage a finding by Shapira et al. (2018) show-0.12 0.17 0.22 0.27 0.32 0.37 0.42 0.47 70 120 170 220 270 320 370 420 470 ROUGE1 R eca ll Word Length Topic 1 Topic 2 L o w e r i n t e r s ec ti on U pp e r i n t e r s ec ti on Figure 2: Example recall-curves of two sessions on an INTSUMM system. ing that a reference summary of a single length can be used to relatively evaluate varying length summaries on a topic with a recall measure such as ROUGE. Thus, utilizing existing MDS datasets is indeed possible for measuring information gain throughout a session's snapshot sequence. Based on this observation we now define three indicators for system performance, first over a single session and then aggregated over all sessions of a system. Per-session indicators. (1) To illustrate the gradual information gain along a session we adopt a recall-by-length curve (Kelly and Lin, 2007; Lin, 2007), see for example Figure 2. The curve's x-axis is the snapshot word-length, chosen as the dominant factor affecting quality, as opposed to number of queries or interaction time, which are not necessarily comparable between sessions. The y-axis is a summary content recall score, such as ROUGE-recall against constant reference summaries. 2 For session  with snapshots  0 ,  1 , ...,  |  | , each  i with word length x i and content recall score y i is plotted on the graph at ( x i , y i ) . (2) We consider the area under the recall-curve (AUC). Intuitively, it is desirable for an INTSUMM system to generate more salient information earlier: assuming salient information is more relevant to users, this property means interaction is ceased sooner, as soon as the information needs are met. Accordingly, AUC is higher when content is more relevant and is retrieved earlier. AUC is defined between start and end x-values, fixed for comparable 2 Any standard summary content recall measure can be used as long as it is consistent, including, e.g., manual mechanisms like Pyramid or nugget-style scoring (Nenkova and Passonneau, 2004; Lin and Demner-Fushman, 2006). measurement (see Figure 2), with y-value scores interpolated at these limits when a curve does not have a snapshot at the specific length(s). (3) We consider the Score@Length metric that reports a score, such as standard ROUGE F 1 , at pre-specified word-lengths, and demonstrates the informational effectiveness of a system at those lengths. This metric enables fair comparison to static summaries at the specified lengths. The inverse Length@Score measure is also examined, and detailed further in Appendix C. Aggregated indicators. The average recall curve , illustrating overall gradual information gain, is computed from individual session recall-at-length curves by interpolating y-values at constant x-value increments and averaging correspondingly. E.g., Figure 3.  [ P. 1] is the average AUC computed from the individual session AUCs by first averaging per topic and then averaging the results over all topics, to give equal weight to each topic. [ P. 2] is the average Score@Length computed similarly to average AUC from individual session Score@Lengths. Automatic evaluation is convenient for fast assessment and consistent comparison, however manual appraisal more accurately forecasts the quality of a summarization system (Owczarzak et al., 2012). Thus, using manual metrics alongside automatic ones is important despite the higher cost it incurs. Our evaluation framework allows doubly leveraging the involvement of human users by asking them to rate different system aspects during the session. We propose the following rating layout, with each measure being scored on a 1-to-5 scale. [ R. 1] After reading the initial summary, the user rates how informative it is for the given topic. This resembles the DUC manual summary content responsiveness rating (Dang, 2006). [ R. 2] To measure the information gain throughout the session, the user rates how much useful information each interaction's response adds. As this rating is scored per interaction, the session average measures overall ability to expose interesting information. [ R. 3] After the session, the user rates how generally well the system responded to the requests throughout the session. [ R. 4] As all human-involved systems should measure perceived usability, the user rates the two UMUX-Lite (Lewis et al., 2013) questionnaire statements: [ R. 4 a ] the system's capabilities meet the requirements and [ R. 4 b ] the system is easy to use. The UMUX-Lite score is a function of these two scores (although they are separately useful) and shows high correlation to the popular, and longer, SUS questionnaire (Brooke, 1996), thus offering a cheaper alternative. Similarly to our automatic measures, these ratings are collected separately per session and then averaged, first per topic and then over all topics, to obtain comparable system scores. The evident advantages of our proposed evaluation framework are: (1) scores are absolute and comparable from one session/system to another; (2) our framework fundamentally and conveniently extends upon prevailing static summarization evaluation practices and utilizes existing standard MDS dataset reference summaries. The evaluation of interactive systems requires real user sessions , as explained in 3. Using a prototype INTSUMM system, described in 5.1, we conducted several cycles of session collection which uncovered multiple user-related challenges, in line with previous work on user task design (Christ-mann et al., 2019; Roit et al., 2020; Zuccon et al., 2013). In particular, recruited users may make undue interactions due to insincere or experimental behavior, yielding noisy sessions that do not reflect realistic system use. Additionally, without an objective informational goal, a user interacts with the system according to subjective interests, producing sessions that are objectively incomparable. Controlled crowdsourcing method. Employing experts to use an interactive system in a user study is usually unnecessary and hinders scalability and accessibility for researchers, making crowdsourcing an appealing and less expensive alternative. While crowdsourcing is ordinarily used for annotation jobs, we show its suitability for system session collection. We designed a three-stage controlled crowdsourcing protocol that mitigates the aforementioned session collection challenges, while filtering out unsatisfactory workers (further details in Appendix B). efficiently filter out insincere workers, and, conversely, discover workers with an ability to apprehend salient information within text. The second stage assigns practice tasks that familiarize the workers to the INTSUMM system interface to prevent experimentation in the actual sessions to be evaluated. Here, the users are also presented with a grounding use-case , or cover-story' as termed by Borlund (2003). The use-case states an objective common goal to follow in interacting with the system, to minimize the effect of subjective preferences, and allow comparison against respective reference summaries with a similar objective goal. An example use-case to follow, applied in our experiments, is produce an informative summary draft text which a journalist could use to best produce an overview of the topic. The use-case is strongly emphasized during practice sessions with integrated guidelines. Workers completing two practice assignments with predominantly relevant interactions are invited to continue on to the final stage. The evaluation session collection stage involves interacting with the evaluated system, for a minimum amount of time per session (e.g., 150 seconds in our experiments), to produce a summary on a topic in light of the same assigned use-case as in the practice stage. Each worker may explore a topic once, and the overall goal is recording sufficiently many sessions per combination of system and topic. Generally in interactive tasks, systems are manually examined over a rather small number of instances (e.g. topics), with only a few users per instance, due to the high cost and complexity of collecting such sessions with experimenters. For example, Christensen et al. (2014) assessed their system on 10 topics, and the ciQA benchmark (Kelly and Lin, 2007) had 6 topics per tested subtask. Our session collection technique provides a more scalable approach, facilitating larger collection processes (e.g., in our experiments in 5 we used 20 topics and  3 sessions per system per topic). We note that, in use cases or domains where experts are required, the proposed three-stage session collection protocol is still fully relevant. It is not limited to the crowdsourcing setting, and can be applied within controlled user-studies if needed. Wild versus controlled crowdsourcing. We illustrate the benefit of the controlled crowdsourcing procedure described above by comparing its results with a wild crowdsourcing preliminary experiment. The latter experiment applied basic worker-Measure Controlled Wild # interactions 12.3 7.0 Approx. filtering ( 99% approval rate and 1000 approved assignments on Amazon Mechanical Turk 3 (AMT)) and did not apply the trap and practice tasks. For quality control, a post-session questionnaire was assigned in order to catch insincere workers. Analysis of the collected sessions showed a substantial improvement in querying behavior in controlled crowdsourcing over sincere wild crowdsourcing (filtering out the insincere wild crowdworkers) the former scored higher than the latter on every evaluation metric. Table 1 presents some qualitative indications of this improvement: controlled users were more engaged (more iterations and more time exploring) and put more thought into their queries (more free-text queries and less suggested queries). Notably, unlike uncontrolled crowd-workers, controlled workers were able to do better than a comparable fully automated baseline, evident from the last table row: the percent difference in ROUGE-1 AUC score from a lower bound simulated baseline (explained in 5.3), is positive (better) for controlled sessions and negative (worse) for wild ones. Finally, the queries of controlled users almost exclusively adhered to the use-case and the many helpful comments from the workers indicated their attentiveness to the task (see Appendix C). We carried out experiments that assess our full evaluation framework and demonstrate its utility. As the few existing INTSUMM systems were not readily available or suitable for adaptation to our experimental setup, we developed an INTSUMM system of our own, shown in Figure 1, with two different algorithmic implementations for comparison. We gathered user sessions with our controlled crowdsourcing procedure and evaluated their quality with 3 https://www.mturk.com our defined measures. We developed a web application, enabling session collection with real users, that follows the INTSUMM schema described in 3: for an input document set, it first presents an initial summary, and then iteratively outputs a summary expansion response per given user request. Specifically, our application supports interactive requests in the form of textual queries from user free-text, summary span highlights and system suggested queries. A system response aims to simultaneously maximize relevance to the query and salience, while refraining from repeating previously presented contents. An initial version of the application was assessed via a small-scale user study of 10 users, with an SUS questionnaire (Brooke, 1996) and the think-aloud protocol (Lewis, 1982) for feedback. Figure 1 displays the improved web application, on the topic Native American Challenges. The left screenshot shows the initial summary with user rating [ R. 1] in [1], an example of a free-text query in the query box [2] and the list of suggested queries in [3]. The right screenshot shows the response to the query entered in the first screenshot. [4] reiterates the last submitted query, with the system response and user rating [ R. 2] in [5]. The last query can also be repeated via a button [6], to obtain additional information on that query. Users can highlight a span from the presented summary, to be automatically pasted to the query box. Initial summaries and expansions are extractive and in bullet-style. In accordance to this interaction flow presented, we implemented two back-end algorithm schemes, denoted S 1 and S 2 , to demonstrate comparison of two INTSUMM systems via our evaluation framework. Each implementation consists of three components: (1) the initial summary generation, (2) the query-response generation and (3) extraction of suggested queries from the source documents. All system outputs must comply to required interaction latency standards (Anderson, 2020; Attig et al., 2017), e.g., a few seconds for the initial summary and a few hundred milliseconds for a query response. While we experimented with some more advanced techniques for MDS generation (e.g., Christensen et al., 2013; Yasunaga et al., 2017), sentence representation (Reimers and Gurevych, 2019) and sentence similarity (Zhang* et al., 2020), we found that these are not practical for incorporation within the interactive low-latency setting, or that they could not handle the relatively large document set inputs. Instead, we developed the two back-end schemas described next (with further details in Appendix A). S 1 runs a sentence clustering initial summary algorithm. Query-responses are generated in MMR-style (Goldstein et al., 2000a) based on semantic similarity between query and sentences. Suggested queries are frequent bigrams and trigrams. S 2 uses TextRank (Mihalcea and Tarau, 2004) for both the initial summary and suggested queries, and a query-response generation approach combining semantic and lexical similarity between query and sentences. The two systems enable experimentation on our evaluation framework and, as we show, demonstrates its viability. Moreover, as apparent in our experimental results (5.4 and user feedback in Appendix C), users attest to the real-world usefulness of these systems. Using our framework, including the baseline systems, future work can develop and examine more advanced methods for INTSUMM , accounting for the latency and input-size challenges. Following our controlled session collection procedure from 4, we released the trap task in AMT and found 48 of 231 workers qualified for the second stage, out of which 25 accepted. 10 workers passed the training stage, from which we recruited 8 highly qualified ones. For the third stage, we collected sessions for 20 topics from DUC 2006, on S 1 and S 2 . Each worker could explore 10 different topics on each system, amounting to 160 possible sessions of which 153 were completed (with at least 3 sessions per combination of topic and system). Since S 1 and S 2 share a common frontend application, users were unaware of which system they are exploring on, and the order was randomized. A minimum exploration time constraint of 150 seconds was set. Initial summaries were  75 tokens (average of 85) and responses were two sentences long. The full controlled crowdsourcing process took one author-work-week, and cost $370. In comparison, wild crowdsourcing described in 4 required a couple days' work and $240 (achieving, as discussed, inferior results), and running a non-crowdsourced user-study of the same magnitude would likely require more work time, and cost an estimated $480 (32 net hours of 16 workers at a commonly acceptable $15 hourly wage). Furthermore, the results of a user study would not necessarily be of higher quality (Zuccon et al., 2013). To our judgement, the controlled crowdworkers are more suitable since they fathom the task before choosing to complete it. In a user study, workers are often unaware of the task before commencing, and may not be fully qualified for or desiring of it. In addition to real user experiments, we simulate each of our two systems on scripted query lists. Simulated sessions provide a means for quick development cycles and quality estimation. The first of two query lists, L Sug , is constructed fully automatically: it consists of the top-10 ordered phrases in the system's suggested queries component per topic. This mimics a lower bound user who adopts the simplest strategy, namely, clicking the suggested queries in order without using judgment even to choose among these queries. The second list, L Oracle , consists of 10 randomly chosen crowdsourced summary content units (SCUs) (Shapira et al., 2019) for each of the topics. Since the SCUs were extracted from the reference summaries of the corresponding topics, they mimic a user who searches for the exact information required to maximize similarity to the same reference summaries which we then evaluate against. While this is not necessarily the optimal query list due to the randomized sampling of SCUs for queries, we consider it our (non-strict) upper bound for the sake of experimentation. The two bounds are relative to the system on which the simulations are carried on. Also, for fair comparison to real sessions, the simulation initial summary and response lengths are similarly set at  75 words and two sentences respectively. We next present the results attained on the 153 sessions collected (5.2), with the purpose of analyzing our full evaluation framework. We gain an understanding on the consistency between automatic and human measurement, and on the comprehensiveness of the full set of measures. Figure 3 presents the average recall-curves and corresponding [ P. 1] averaged AUC scores of the S 1 bounds (5.3) and of the user sessions on S 1 and S 2 . AUC is computed between word-lengths 105 to 333 (the maximum intersection of all ses-sions). Table 2 shows [ P. 2] averaged ROUGE-1 180 200 220 240 260 280 300 320 340 Word Length S Controlled S L^Sug 73.7 72.5AUC 73.7AUC 73.8AUC 76.9AUC 40 45 50 55 60 65 70 75 80 0.18 0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.4 0.42 0.44 0.46 0.48 100 125 150 175 200 225 250 275 300 325 AUC @ 105 333 ROUGE 1 R eca ll Word Length S L^Oracle S Controlled S Controlled S L^Sug Figure 3: The average recall-curves, along with corresponding AUC scores (unrelated to the x-axis) and their confidence intervals (  95% ), of the upper and lower bound sessions and of user sessions of the two systems. based Score@Length. Scores rank consistently on ROUGE-2, ROUGE-L and ROUGE-SU (see Appendix C). It is evident from Figure 3 and Table 2 that the results on collected sessions indeed fall between the two bounds in all measures. This demonstrates the effectiveness of interactive summarization, even when using relatively simple algorithms: the algorithm enables fast information processing of input texts, and users effectively direct the algorithm to salient areas. Additionally, the scores of S 1 and S 2 are close, providing no significant insights when comparing these two systems, which is surprising due to their distinct implementations. Manually reviewing the results, we were convinced that the systems indeed happen to perform at similar quality overall . However, when assessing the systems' separate components and inspecting user-provided ratings, we gain awareness of some interesting distinctions. the ratings provided by the users. The initial sum-maries' ROUGE-1 F 1 scores are computed against the reference summaries, with a slight advantage for S 1 over S 2  similar to the users' initial summary ratings. For the query-response component, we compute the average ROUGE F 1 score of the independent responses to the queries in L Oracle , against the reference summaries. Again, user ratings reflect a similar trend that the query-response component of S 2 slightly outscores that of S 1 . Overall we see that S 1 provides a better initial summary while S 2 handles queries better. Also, users tend to be more satisfied by S 2 , likely due to its ability to respond better to queries. This claim is evident from the positive correlation between [ R. 3] and [ R. 4 a ] , r = 0 . 68 , p < 0 . 001 in S 1 and r = 0 . 63 , p < 0 . 001 in S 2 . In terms of absolute UMUX-Lite scores [ R. 4] , 68 is considered average, and above 80 is considered excellent, meaning both S 1 and S 2 got high usability scores, with a preference for S 2 . An additional analysis finds a positive correlation between per-iteration response [ R. 2] scores and the relative per-iteration increase in ROUGE recall (e.g. for ROUGE-1 r = 0 . 36 , p < 0 . 001 in S 1 and r = 0 . 33 , p < 0 . 001 in S 2 ), hinting at the credibility of correlation between human ratings and relative increase in ROUGE within sessions. To conclude, our findings are favorable in terms of the framework's internal consistency of measures and soundness of the computed scores. For a more conclusive appraisal of the full evaluation framework, additional systems are to be run through the process, regardless of the accidental similarity between our two baselines. We proposed a comprehensive evaluation framework for user-guided expansion-based interactive summarization  a vital ingredient for the methodological advancement of interactive summarization research which was unaccounted for until now. Our controlled crowdsourcing procedure makes INTSUMM system session collection accessible, scalable and replicable. The evaluation measures in our framework provide a thorough assessment with absolute scores that enable comparison of INTSUMM systems. Our framework provides the means to advance INTSUMM research on system development and improved evaluation. All solutions, including our implemented baseline systems, are publicly available to enable comparison of new INTSUMM systems to ours on any MDS dataset. In future work, it is worthwhile to separately assess the effectiveness of individual interaction modes, including ones incorporated in our implementation and others, e.g., full questions input by users. These would require further experimentation, additional evaluation metrics, and the possible use of datasets from tasks other than MDS. Within our expansion-based framework, we can consider additional measures of textual consistency, coherence, and relevance of responses to queries. We may also test additional approaches for summarization: e.g., abstractive summarization for flexible synthetic summary generation, requiring further evaluation of factuality and truthfulness. Beyond our framework, that targets objective quality, INTSUMM systems should also be evaluated according to their compatibility with personalized, subjective use. User-study. Our system-testing user-study (men-tioned in 5.1) was conducted on a university campus, and students within different age groups and from different backgrounds were recruited through a social media group for hiring for experiments and user studies. We required a high level of English for participation. People were accepted until the required amount of participants (10) was reached, without any targeted filtering. An individual study lasted around 30 minutes for a payment of around $10. Crowdsourcing. There were several rounds of crowdsourcing, with varying tasks. Due to the need for fluent English speaking workers, a location filter was set on the AMT platform for English (as primary language) speaking countries. At least one of the authors tested each task before its release to estimate worst-case task completion duration. The payment was then set according to $9 per hour for the estimated required time. In practice, almost all tasks were completed in less than the time estimated, and payment was well above $9 per hour. Very few assignments were rejected in cases of clear insencereness (unreasonably fast submission or senseless behavior). Dataset usage. As pointed out throughout the paper, the DUC 2006 dataset was utilized. It was obtained through the required means on the DUC website ( duc.nist.gov ). There was no possibility to reconstruct the dataset (document sets and reference summaries) within any of the conducted user study and crowdsourcing tasks. Application. Our INTSUMM systems' outputs are extracts from the input document sets. As described in Appendix A, the algorithms for initial summary and query-response generation do not contain any intentional biasing. The intended purpose of any INTSUMM system is to allow readers to make sense of large bodies of text through assisted exploration. Future work may open the door to more personalized algorithms and abstractive outputs. This would require extra care in making sure systems are ethically sound by adding targeted evaluation measures. Compute time. As emphasized in the paper, INTSUMM systems require low latency and are hence relatively computationally cheap. During our research we ran some algorithms, to test for our systems, that required up to several hours of compute time per run, on a standard server. We would like to thank Guiseppe Carenini for his helpful advice, and the anonymous reviewers for their constructive comments. This work was supported in part by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grants DA 1600/1-1 and GU 798/17-1); by the BIU Center for Research in Applied Cryptography and Cyber Security in conjunction with the Israel National Cyber Bureau in the Prime Minister's Office; by the Israel Science Foundation (grants 1157/16 and 1951/17); by a grant from the Israel Ministry of Science and Technology; by the NSF-CAREER Award #1846185; and by a Microsoft PhD Fellowship.
abstain abstain objective method abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain method objective objective abstain objective abstain result objective abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method method method objective abstain abstain method abstain objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other