We introduce SUMMSCREEN , a summarization dataset comprised of pairs of TV series transcripts and human written recaps. The dataset provides a challenging testbed for abstractive summarization for several reasons. Plot details are often expressed indirectly in character dialogues and may be scattered across the entirety of the transcript. These details must be found and integrated to form the succinct plot descriptions in the recaps. Also, TV scripts contain content that does not directly pertain to the central plot but rather serves to develop characters or provide comic relief. This information is rarely contained in recaps. Since characters are fundamental to TV series, we also propose two entity-centric evaluation metrics. Empirically, we characterize the dataset by evaluating several methods, including neural models and those based on nearest neighbors. An oracle extractive approach outperforms all benchmarked models according to automatic metrics, showing that the neural models are unable to fully exploit the input transcripts. Human evaluation and qualitative analysis reveal that our non-oracle models are competitive with their oracle counterparts in terms of generating faithful plot events and can benefit from better content selectors. Both oracle and non-oracle models generate unfaithful facts, suggesting future research directions. 1 1 Introduction Abstractive summarization aims to produce a summary that concisely expresses key points of the input document rather than merely extracting pieces of it. Existing datasets are constructed from various domains, such as news (Sandhaus, 2008; Hermann  Work done while the author was at the University of Chicago. [ The apartment ] Sheldon : What color would you like to be ? Leonard : Well , I 'd like to be green , but you know you always take it . Sheldon : That 's not true . Any color 's fine with me . Yeah , I could be a a combination of blue and yellow . Leonard : Blue and yellow make green . Sheldon : Well , then it 's settled . Penny : Hi . Ready to go ? Sheldon : Oh , good news , we ordered lunch , so we can all stay here and play Lord of the Rings Risk . Amy : Sheldon , we said that we would play games with you tonight . Sheldon : Oh , no , we 'll still be playing it tonight , this game can easily take eight hours . Penny : Sweetie , you really thought I 'd want to do this ? Leonard : No . Penny : Well , did you tell him that ? Leonard : Yes . Penny : Did you say it out loud with words ? Leonard : No . Penny : I do n't want to spend the whole day playing a board game . Sheldon and Leonard are happy playing a board game until Amy and Penny say they are tired of doing what the guys want  Transcript: Recap: Figure 1: Excerpts from an example from SUMMSCREEN . et al., 2015; Rush et al., 2015; Narayan et al., 2018; Grusky et al., 2018), online forums (Vlske et al., 2017), meeting dialogues (Janin et al., 2003; Carletta et al., 2005), and webpages (Chen et al., 2020). However, few datasets exist for abstractive summarization of narrative text, which focuses on entities and dialogue among entities, with plot details often communicated indirectly via dialogue. In this work, we build SUMMSCREEN , an abstractive summarization dataset combining TV series transcripts and episode recaps. Figure 1 shows an example from SUMMSCREEN . Several aspects of SUMMSCREEN make it a challenging testbed for abstractive summarization. First, the relationship between character dialogue and plot details is not straightforward. Plot events 8602 are often expressed indirectly in dialogue, and dialogue contains other information that is not directly relevant to the plot, such as character development and humor. Also, a typical episode has multiple subplots that proceed in parallel, with consecutive scenes often describing different subplots. Solving SUMMSCREEN requires drawing information from utterances across a wide range of the input and integrating the information to form concise plot descriptions. Moreover, since actual TV episodes ground their scripts with audio-visual accompaniment, many details may be omitted from the transcript itself. This omission of details and the other challenging aspects mentioned above have inspired research into other NLP tasks on TV show transcripts, such as entity tracking (Chen and Choi, 2016; Choi and Chen, 2018) and coreference resolution (Chen et al., 2017; Zhou and Choi, 2018). Another prominent characteristic of TV series transcripts is their focus on characters. To reflect this aspect, we propose two entity-centric metrics to evaluate the quality of generated plot summaries. One is based on bags of characters, which measures the overlap of the characters that appear in both the generated and reference recaps. The other metric measures character relations: the overlap of cooccurrences of character pairs in generations and recaps. We empirically evaluate several types of methods on SUMMSCREEN . We consider nearest neighbor models, which look up similar transcripts or recaps, neural abstractive summarization models, and hybrid models, which use the nearest neighbor models as content selectors followed by abstractive summarization. Oracle extractive approaches outperform all models on all the automatic metrics. These results suggest that the benchmarked methods are unable to fully exploit the input transcripts and that improving content selection may be a promising research direction. Human evaluations show that our non-oracle hybrid models are competitive with their oracle counterparts in terms of generating faithful plot events. Hybrid models may be promising approaches for future research. Qualitative analysis shows that neural models tend to generate generic summaries, hybrid models can benefit from better content selection, and hybrid models sometimes generate unfaithful details. There has been prior work on extractive screenplay summarization (Gorinski and Lapata, 2015; Papalampidi et al., 2020), and analyzing crime drama (Frermann et al., 2018). The majority of TV show transcripts are dialogues, relating our work to prior work on dialogue and meeting summarization. Relevant datasets have been studied for medical dialogues (Joshi et al., 2020; Krishna et al., 2021), chitchat (SAMSum; Gliwa et al., 2019), podcasts (Clifton et al., 2020), meetings (AMI; Carletta et al., 2005; ICSI; Janin et al., 2003; QMSum; Zhong et al., 2021), livestreams (StreamHover; Cho et al., 2021), online forums (ForumSum; Khalman et al., 2021) and news interviews (MediaSum; Zhu et al., 2021). There have been attempts in summarizing long-form text (other than screenplays), such as books (Mihalcea and Ceylan, 2007), scientific articles (PubMed and arXiv; Cohan et al., 2018), multiple news articles (Multi-News; Fabbri et al., 2019), opinionated text (RottenTomatoes; Wang and Ling, 2016), patents (Sharma et al., 2019), TV show stories (TVRecap; Chen and Gimpel, 2021) and (extractive summarization of) chapters of novels (Ladhak et al., 2020). More detailed discussion on the differences between these datasets and SUMMSCREEN is in the next section. Recently there have been efforts on adapting resources for TV shows for different tasks, including question answering (Ma et al., 2018; Yang and Choi, 2019), speaker identification (Ma et al., 2017), sarcasm detection (Joshi et al., 2016), emotion detection (Zahiri and Choi, 2017; Hsu and Ku, 2018), character relation extraction (Yu et al., 2020), and story generation (Chen and Gimpel, 2021). An instance in SUMMSCREEN contains a transcript from TV series and its corresponding recap. The transcripts consist of dialogue utterances with speaker names, and descriptions of scenes or character actions. The recaps are human-written summaries of the corresponding transcripts. Figure 1 shows an example in SUMMSCREEN from the TV show The Big Bang Theory. The transcript documents a dialogue involving four characters (Sheldon, Leonard, Penny, and Amy) about playing a board game, and the recap summarizes the dialogue into sentences. We use two sources to construct SUMMSCREEN : The TV MegaSite, Inc. (TMS) 2 and ForeverDreaming (FD), 3 both of which provide community-contributed transcripts. As FD does not provide recaps, we obtain recaps of FD shows from Wikipedia and TVMaze. 4 To ensure dataset quality of SUMMSCREEN , we filter out instances based on two criteria. First, the overlap ratio of TV show characters appearing in the recap and its transcript should be higher than 85%. We use this criterion to ensure that the alignments between recaps and transcripts are correct. Second, the number of transcript lines that have speaker information (char-acter utterances) should be larger than 100. We use this criterion to eliminate transcripts that are essentially subtitles, i.e., utterances without speaker information. In practice, for each transcript line, if a colon symbol appears in the first 8 tokens and there exists at least one character name in front of the colon symbol, we will count it as a character utterance. We note that FD and TMS do not have overlapping TV series. In Table 1, we compute n-gram overlap ratios between recaps and transcripts for measuring the abstractiveness of SUMMSCREEN . From the results, We find that despite SUMMSCREEN has longer summaries, its fraction of overlapping four-gram is comparable to XSum (Narayan et al., 2018) which is known for abstractiveness, suggesting that SUMMSCREEN favors abstractive approaches. Table 2 shows data statistics and Figure 2 shows 2 http://tvmegasite.net/ 3 transcripts.foreverdreaming.org 4 www.tvmaze.com , an online TV database curated by TV fans. the genres of the TV shows from the two sources. 5 When computing the number of unique characters in TV shows, we first collect the character names from TVMaze and the named entities 6 preceding the colon symbols in transcripts. We then perform string matching to obtain numbers of TV show characters in recaps and transcripts. From these two tables, we observe that FD and TMS are different in many aspects. First, FD covers more diverse genres than TMS. This is partly due to the fact that TV shows from TMS are soap operas. Second, transcripts from FD are longer, which is caused by the fact that the transcripts from FD tend to have more descriptions about environments or character actions, whereas the ones from TMS are mostly 5 The genre information is from TVMaze where a TV show may correspond to multiple genres. made up of dialogue (see Table 2). Third, recaps from FD are shorter whereas recaps from TMS seek to cover more details. Fourth, writing styles are more diverse in FD than those in TMS. In light of these differences, we treat FD and TMS as different datasets in the following experiments. We create train/dev/test splits for FD and TMS by ensuring the ratio to be roughly 10:1:1, and filter out instances in the dev/test splits if the reference texts are shorter than 30 word tokens. The statistics of the splits are shown in Table 3. 3.2 Dataset Comparison We compare SUMMSCREEN to other abstractive dialogue summarization datasets in Table 4. SUMMSCREEN differs from other datasets in several ways: 1. Compared to recently proposed large-scale dialogue summarization datasets (i.e., SAMsum and MediaSUM), SUMMSCREEN has longer source inputs. 2. Compared to other dialogue summarization datasets, SUMMSCREEN has larger numbers of speakers per instance. The TV series genre focuses on narrative, which is typically entity-centric and can include multiple parallel subplots in a single episode. 3. Compared to AMI, ICSI and QMSum, which are long-input meeting summarization datasets, SUMMSCREEN has far more instances. 4. Unlike most of the other datasets, SUMMSCREEN contains many episodes of a single show (e.g., more than 3k episodes for TMS). This episodic structure could be used to model character arcs, the evolution of character personality traits and character relationships over episodes, among others. Properties (1) and (2) above make extracting information from transcripts more challenging than other datasets. The third property means that SUMMSCREEN is large enough to train and evaluate neural methods. The Spotify Podcast Dataset (Clifton et al., 2020) and StreamHover (Cho et al., 2021) are similar to SUMMSCREEN in that they contain transcribed speech and summaries. However, the transcriptions are obtained automatically and therefore contain errors. 7 The datasets therefore involve speech processing (or at least handling speech recognition errors) compared to SUMMSCREEN , which has human-written transcripts. Since MediaSum is constructed from news transcripts, it is the most similar dataset in Table 4 to SUMMSCREEN . However, the summaries in MediaSum are twenty times shorter than those in SUMMSCREEN , and the average number of speakers per instance is only a quarter of that in SUMMSCREEN . Furthermore, our results in Sec. 5.2 indicate that our dataset is much harder than MediaSum as the pretrained models perform worse on our dataset than on MediaSum according to automatic metrics. More detailed analysis is in the next subsection. In this subsection, we qualitatively analyze the challenging aspects of SUMMSCREEN . Since the transcripts focus on dialogue among characters, along with limited descriptions of scenes and actions, it leads to the challenge that plot information is not stated explicitly but rather only implied in the dialogue. For example, the transcript in Figure 1 does not explicitly describe what Sheldon and Leonard are playing. However, it is implied by Sheldon when he mentions playing Lord of the Rings Risk, and later by Penny when she says that she does not want to spend the whole day playing a board game. A related challenge is the need to understand the context in which characters' utterances are situated. In the example, the recap describes four characters taking sides regarding playing a board game. The transcript expresses the characters' sentiments through their interactions with one another. The conflict does not occur until Sheldon proposes to stay here and play Lord of the Rings Risk, and it becomes more apparent when Penny mentions she does not want to play the board game. Given the context, Leonard's series of yes and no responses to Penny's questions is largely due to the awkward sit-7 For this reason, we do not include their statistics in Table 4. 8605 # instances # tokens (input) # tokens (summary) # speakers Domain Multi-News 56.2k 2103.5 264.7 News RottenTomatoes 3.7k 2124.7 22.2 Reviews arXiv 215k 4938.0 220.0 Science PubMed 113k 3016.0 203.0 Science GovReport 19.5k 9409.4 553.4 Government Reports TVRecap 29.0k 1868.7 221.6 Television Series SAMSum 16.4k 83.9 20.3 2.2 Chitchat ForumSum 4.1k 303.5 36.0 6.7 Forum Messages MediaSum 463.6k 1553.7 14.4 6.5 News Interviews AMI 137 4757.0 322.0 4.0 Meetings ICSI 59 10189.0 534.0 6.2 Meetings QMSum 1.8k 9069.8 69.6 9.2 Meetings SUMMSCREEN 26.9k 6612.5 337.4 28.3 Television Series Table 4: Statistics for datasets focusing on abstractive summarization for long-form text or dialogue. uation, and it actually shows that he is happy playing the game as he and Sheldon are doing so at the beginning of the scene. Similarly, Amy mentions their previous agreement with Sheldon as a way of politely declining Sheldon's plan. The sentiments of characters are not necessarily easily discernible from their utterances but rather must be inferred using context and knowledge about the characters. Another challenge in SUMMSCREEN is the need to draw information from a wide range of the input transcripts, which arises for two primary reasons. First, there are many utterances that serve a purpose other than driving the plot forward. They may help to develop characters or character relationships, or to add humor or suspense. These lines enrich the narrative but their information content is often omitted from the summaries. For example, in the first instance in Figure 3, we show key lines from the transcript that pertain to the excerpt of the summary. There are many other lines between the lines shown, which are conversations between the doctor and other characters. This property necessitates the models' ability to correctly attend to major events across the transcript when generating summaries. The pattern can also be observed in Table 2 through the differences between the number of unique characters in recaps and transcripts. More than half of the characters in the transcripts are not contained in the recaps. The second reason why information needs to be combined across wide ranges of the input relates to scene breaks and multiple plots. As a TV show often narrates a few plots in parallel, scene breaks are used to separate the stories. The discontinuity sometimes requires models to connect subplots hundreds of lines apart. For example, for the second instance in Figure 3, the show uses scene breaks to express what is happening when Cullen 8606 Bohannon escapes from the Swede, which is why there are almost two hundred lines between Cullen Bohannon's escape and his appearance at Durant's office. In this section, we describe modeling approaches that we benchmark on SUMMSCREEN . We note that since the meaning of sentences in transcripts is highly context-dependent, extractive summarization approaches are not expected to be useful for this dataset. We report the results from nearest neighbor-based extractive summarizers mostly for characterizing the dataset. We use transformer based sequence-to-sequence architectures (Vaswani et al., 2017). Because transcripts are quite long, We limit the number of encoder hidden vectors that are used in the decoder's attention mechanism. To do so, when encoding transcripts, we first append a special token [EOS] to each line of the transcript, and then linearize the transcript. We then only feed the vectors representing these special tokens to the decoder. We use the Longformer (Beltagy et al., 2020) as our encoder architecture, and set the [EOS] tokens to use global attention. For our decoders, we use the standard transformer architecture. We consider two metrics when finding nearest neighbors: BM25 (Robertson et al., 1995) (a popular metric for information retrieval), and ROUGE scores (Lin, 2004). We use ROUGE scores as they are used for evaluation, and we use BM25 because it is designed for retrieving long documents whereas ROUGE scores are not. When using ROUGE scores, we use the average of ROUGE-1, ROUGE-2, and ROUGE-L. We consider three types of nearest neighbor search: transcript-to-transcript, recap-to-transcript, and recap-to-recap. Recap-to-transcript (NNM-r2t). We use each sentence in the recap as queries and the lines in the corresponding transcript as candidates. The generation is formed by the nearest neighbors for each sentence. We use BM25 or ROUGE scores as the metric. This method can serve as an oracle result for an extractive summarization system, showing roughly how much information can be extracted at the utterance level from the source transcript. Transcript-to-transcript (NNM-t2t). We use the transcripts in the test sets as queries, the transcripts in the training sets as candidates, and then find the nearest neighbors using BM25. The generations are the corresponding recaps. This baseline measures the similarity of instances between training and test splits. Recap-to-recap (NNM-r2r). This setting is similar to the transcript-to-transcript setting, but we use recaps for both queries and candidates, and we use ROUGE and our proposed entity-centric scores (see Sec. 5.1 for more details) as the metric. When using the entity metrics, we use the average of the 4 metric scores. This is an oracle baseline of the transcript-to-transcript setting and also measures the similarity of the splits. As content selection has been shown to be helpful in prior work (Gehrmann et al., 2018; Liu et al., 2018), we use the recap-to-transcript nearest neighbor model and BM25 as the metric to select the most salient content from transcripts, and then apply neural models to the selected content when performing generation. As these methods combine nearest neighbor models and neural models, we refer to them as hybrid models. In particular, for each sentence in the recap, we find the top three most similar lines in the transcript, include two extra lines that come before or after the selected lines as context, and also include a line that is retrieved by using the whole recap. As the selected content is significantly shorter than the original transcript, it allows us to use pretrained models. 8 Therefore, in this setting, we fine-tune a pretrained BART-large model (Lewis et al., 2020). We note that as the nearest neighbor models rely on the gold standard recaps, this hybrid model demonstrates an approximate upper bound on performance when using powerful content selectors. 9 To establish a non-oracle baseline, we train neural models to predict the selected lines, and then fine-tune BART-large models on the predicted lines. Details of the architecture for this component, which we call our neural content selector, are in the appendix. 8 After the selection steps, the average number of tokens of the transcripts for FD and TMS reduces to 1138.9 and 3252.7 respectively. 9 We use the maximum sequence length of 1024 (i.e., we truncate the input sequences if they are longer than 1024) for BART-large due to computational constraints. We report BLEU (Papineni et al., 2002), ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-L (RL). We report the average of these four metrics as it generally shows the semantic similarities between generations and references. We will refer to these metrics as generic metrics as they treat each word equally. As characters are fundamental to TV show plots, we believe the quality of plot summaries also depends on including the right characters. To take this factor into account, we compute several bag of character (BoC) metrics based on the fraction of the overlapping characters between generated and gold standard recaps. Formally, we define the BoC precision to be | f ( generation )& f ( r ) | | f ( generation ) | (1) where f is a function that extracts the bag of characters from some text, where we perform string matching based on the character names that are automatically extracted during dataset construction (see Sec. 3.1), & computes the intersection of two bags, |  | returns the size of its inputs, and r is the gold standard recap. Similarly, we define the BoC recall to be | f ( generation )& f ( r ) | | f ( r ) | (2) Since BoC does not consider relations between characters, we also report bag of character relations (BoR) metrics based on the cooccurrence of character pairs. We assume two characters are related when they appear in the same sentence. After obtaining the character relations from the gold standard recaps and the generations, we compute recall and precision against the recaps following the same approach as BoC. We note that the extracted relations are non-directional, and BoR does not consider frequency of the cooccurrences. We also report the averages of both precisions and recalls from both the BoC and BoR metrics. We report test results for FD and TMS in Table 5. Our findings for the nearest neighbor models are as follows: 1. We find that the nearest neighbor models give strong performance on our dataset. In particular, NNM-r2t shows the best performance in general. This demonstrates that there is still room for improving the ability of our neural models to extract the most useful information from transcripts, suggesting that improved transcript modeling may be a fruitful research direction for these datasets. 2. We observe that NNM-r2r exhibits different strengths when based on different metrics, for example, using ROUGE scores will lead to results favorable to generic metrics. As for the results involving neural models, our findings are as follows: 1. The neural model shows strong performance in generic semantic matching but it is relatively weak in entity metrics compared to the non-oracle baselines. (see the appendix for more discussion). 2. The hybrid model is better than the neural model in terms of generating character mentions and relations. With the help of the oracle content selector, the hybrid model improves significantly in both semantic matching and entity-related metrics, showing that future research may find improvement by designing better content selectors. We study the effect of transfer learning using these two resources. When doing so, we use the training and development sets constructed from both resources, and at test time, we evaluate models on the official test splits. We experiment with the oracle hybrid model and report results in Table 6. In general, we find that extra training data helps FD. We hypothesize that this is due to the relatively small size of FD. However, for TMS, training on FD harms performance, which is likely because of the larger training set size for TMS and the differences between the two resources. We conduct human evaluations for three models: NNM-t2t, hybrid model, and hybrid model (oracle). To evaluate two key aspects of SUMMSCREEN , namely events and characters relationships, we ask human annotators two questions. The first is Do 8608 Generic Metrics Entity Metrics BLEU R1 R2 RL avg. BoC-p BoC-r BoR-p BoR-r avg. ForeverDreaming NNM-r2t (oracle, BM25) 3.4 34.3 6.6 29.6 18.5 70.5 61.9 36.4 16.1 46.2 NNM-r2t (oracle, RG) 3.9 34.8 8.5 31.5 19.7 76.7 63.3 46.5 21.3 52.0 NNM-r2r (oracle, RG) 9.9 38.8 11.5 33.9 23.5 50.6 51.4 24.6 26.8 38.4 NNM-r2r (oracle, Entity Metrics) 5.5 31.1 6.8 27.1 17.6 58.6 79.6 26.4 43.7 52.1 NNM-t2t 7.9 31.3 7.8 27.4 18.6 56.5 59.2 28.2 29.4 43.3 Neural model 2.6 25.9 4.2 23.8 14.1 54.7 38.5 22.8 15.1 32.8 Hybrid model 2.4 25.3 3.9 23.1 13.7 61.2 51.4 29.8 23.6 41.5 Hybrid model (oracle) 3.0 26.4 5.0 23.3 14.4 70.0 57.8 36.9 29.1 48.5 TVMegaSite NNM-r2t (oracle, BM25) 6.7 45.0 10.2 43.0 26.2 82.5 80.4 57.7 18.1 59.7 NNM-r2t (oracle, RG) 8.5 44.1 11.7 42.4 26.7 85.2 76.8 61.2 16.9 60.0 NNM-r2r (oracle, RG) 7.9 49.0 11.6 46.9 28.9 59.2 59.0 29.5 29.9 44.4 NNM-r2r (oracle, Entity Metrics) 4.9 42.8 8.8 40.4 24.2 60.8 81.7 26.0 37.5 51.5 NNM-t2t 6.2 43.2 8.6 41.4 24.9 63.2 69.3 31.8 35.3 49.9 Neural model 7.9 42.9 11.9 41.6 26.1 86.1 48.7 48.9 22.3 51.5 Hybrid model 5.5 38.8 10.2 36.9 22.8 84.5 57.2 51.0 29.3 55.5 Hybrid model (oracle) 8.9 42.1 11.9 40.9 25.9 84.0 69.5 56.4 36.8 61.7 Table 5: Results on the SUMMSCREEN test sets. BLEU, R1, R2, and RL are BLEU and ROUGE scores between model generated and reference recaps. Bo{C,R}-{p,r} are precision and recall for bag of characters and bag of character relations, respectively. The highest numbers for each dataset in each column are in bold. Generic Entity ForeverDreaming FD Only 16.5 47.3 TMS + FD 16.9 50.1 TVMegaSite TMS Only 25.9 61.7 TMS + FD 23.2 58.0 Table 6: Results of the oracle hybrid model comparing training on both datasets (TMS + FD) to training on the in-domain dataset only. The metrics are averaged scores of the generic and entity metrics. Training on both datasets helps for FD but hurts for TMS. Predicates Character Relation NNM-t2t 1.6  0.8 2.1  1.1 Hybrid model 2.3  0.9 2.0  1.0 Hybrid model (oracle) 2.4  1.0 2.4  1.0 Table 7: Human evaluation results. We report the average scores and their corresponding standard deviations for questions on predicate match and character relation similarity. the predicates in the generation match the predicates in the reference? 10 The second is When multiple characters are mentioned as being related in some way in the generated recap, are those same characters mentioned as being related in some way in the reference? We disregard the subjects in the first question because the second question involves evaluating characters and we want the two questions to focus on different aspects to maximize the 10 By predicate here we mean the part of a sentence or clause containing a verb and stating something about the subject (e.g., went home in John went home). efficiency of human annotations. Ratings are on a 1-5 scale with 5 indicating a perfect match. We randomly picked instances from the FD test set. We (the authors) annotated 120 instances in total for each question. After dropping 2 invalid annotations for the second question (as there may not be multiple characters mentioned), we summarize results in Table 7. While trends for the model performance on character relations are generally similar to our observations in Table 5, the results for predicate match are very different for NNM-t2t. This is likely because the first question is about predicates disregarding the correctness of the participants. We also want to highlight that compared to the oracle hybrid model, the non-oracle one shows competitive performance on predicate match but is less close in terms of generating correct character relations, showing future opportunities for improving this model. In Table 8, we show generation samples for the following models: the neural model, the hybrid model, and the oracle hybrid model. The neural model manages to fit most of the character names from the reference into the generation. The generation shares similar topics with the reference, but compared to the hybrid models it lacks specifics. This matches our observations from the automatics metrics where the neural model performs better on the generic metrics but worse on the entity metrics on the non-anonymized datasets. We hypothesize 8609 Reference The remains of two witches , one of which is from the Salem witch trials from the 1600s and the other a modern day Wiccan , are discovered in the remains of a burnt out cabin . that this is caused by the difficulty of modeling long-form text. In the output of the non-oracle hybrid model, many facts that are not mentioned in the reference are actually from the transcript. For example, 40-year-old woman and de-fleshed prior to her death are in the transcript. Despite containing many specifics, the generation misses a few important details, such as the absence of mentioning main characters involved (i.e., Brennan and Booth). It also has incorrect facts. For example, according to the transcript, there are rocks at the scene, but the model describes the setting as a rock quarry. Compared to the other three models, the generation from the oracle hybrid model is the most faithful, although there are still incorrect facts (e.g., ... and they are having a hard time adjusting to their new life in the city.). The differences between the oracle and non-oracle hybrid model suggest that future research can focus on improving models' capabilities of doing content selection. As both oracle and non-oracle hybrid models suffer from generating incorrect facts, faithfulness in generation is also an important future research direction. We construct SUMMSCREEN , which contains pairs of TV show transcripts and recaps. We qualitatively analyze the challenging aspects of our dataset. We propose two entity-centric metrics to evaluate generated summaries with one focusing on character overlap and the other focusing on overlap of character pairs. Empirically, we benchmark several neural models and nearest neighbor models for characterizing our datasets, finding that an oracle extractive summarizer gives the strongest performance according to the automatic metrics. Human evaluations show that the non-oracle hybrid model is competitive at generating faithful topics. Qualitative analysis shows that the hybrid model can benefit from better content selectors and both oracle and non-oracle hybrid models suffer from generating inaccurate details, highlighting several directions for future research. We wish to thank The TV MegaSite, Inc. and Forever Dreaming for allowing us to use and redistribute their data for research purposes. This work was supported in part by a Google Fellowship to M. Chen.
abstain abstain abstain abstain abstain abstain objective method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method method abstain abstain result abstain abstain other abstain other other other other abstain abstain abstain abstain abstain objective result method abstain method abstain method objective abstain method result other abstain objective result result abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method method method method method method method method method abstain method abstain method abstain abstain method abstain abstain abstain method method abstain method method method method method method method method abstain method method method method method method method method method method method method result abstain result abstain abstain result result abstain abstain abstain abstain method method abstain abstain abstain abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method objective result abstain abstain other other