Suspense is a crucial ingredient of narrative fic-tion, engaging readers and making stories compelling. While there is a vast theoretical literature on suspense, it is computationally not well understood. We compare two ways for modelling suspense: surprise, a backward-looking measure of how unexpected the current state is given the story so far; and uncertainty reduction, a forward-looking measure of how unexpected the continuation of the story is. Both can be computed either directly over story representations or over their probability distributions. We propose a hierarchical language model that encodes stories and computes surprise and uncertainty reduction. Evaluating against short stories annotated with human suspense judgements, we find that uncertainty reduction over representations is the best predictor, resulting in near human accuracy. We also show that uncertainty reduction can be used to predict suspenseful events in movie synopses. As current NLP research expands to include longer, fictional texts, it becomes increasingly important to understand narrative structure. Previous work has analyzed narratives at the level of characters and plot events (e.g., Gorinski and Lapata, 2018; Martin et al., 2018). However, systems that process or generate narrative texts also have to take into account what makes stories compelling and enjoyable. We follow a literary tradition that makes And then? (Forster, 1985; Rabkin, 1973) the primary question and regards suspense as a crucial factor of storytelling. Studies show that suspense is important for keeping readers' attention (Khrypko and Andreae, 2011), promotes readers' immersion and suspension of disbelief (Hsu et al., 2014), and plays a big part in making stories enjoyable and interesting (Oliver, 1993; Schraw et al., 2001). Computationally less well understood, suspense has only sporadically been used in story generation systems (O'Neill and Riedl, 2014; Cheong and Young, 2014). Suspense, intuitively, is a feeling of anticipation that something risky or dangerous will occur; this includes the idea both of uncertainty and jeopardy. Take the play Romeo and Juliet: Dramatic suspense is created throughout  the initial duel, the meeting at the masquerade ball, the marriage, the fight in which Tybalt is killed, and the sleeping potions leading to the death of Romeo and Juliet. At each moment, the audience is invested in something being at stake and wonders how it will end. This paper aims to model suspense in computational terms, with the ultimate goal of making it deployable in NLP systems that analyze or generate narrative fiction. We start from the assumption that concepts developed in psycholinguistics to model human language processing at the word level (Hale, 2001, 2006) can be generalised to the story level to capture suspense, the Hale model. This assumption is supported by the fact that economists have used similar concepts to model suspense in games (Ely et al., 2015; Li et al., 2018), the Ely model. Common to both approaches is the idea that suspense is a form of expectation: In games, we expect to win or lose instead in stories, we expect that the narrative will end a certain way. We will therefore compare two ways for modelling narrative suspense: surprise, a backward-looking measure of how unexpected the current state is given the story so far; and uncertainty reduction, a forward-looking and measure of how unexpected the continuation of the story is. Both measures can be computed either directly over story representations, or indirectly over the probability distributions over such representations. We propose a hierarchical language model based on Generative Pre-Training (GPT, Radford et al., 2018) to encode story-level representations and develop an 1764 inference scheme that uses these representations to compute both surprise and uncertainty reduction. For evaluation, we use the WritingPrompt corpus of short stories (Fan et al., 2018), part of which we annotate with human sentence-by-sentence judgements of suspense. We find that surprise over representations and over probability distributions both predict suspense judgements. However uncertainty reduction over representations is better, resulting in near human-level accuracy. We also show that our models can be used to predict turning points, i.e., major narrative events, in movie synopses (Pa-palampidi et al., 2019). In narratology, uncertainty over outcomes is traditionally seen as suspenseful (e.g., O'Neill, 2013; Zillmann, 1996; Abbott, 2008). Other authors claim that suspense can exist without uncertainty (e.g., Smuts, 2008; Hoeken and van Vliet, 2000; Gerrig, 1989) and that readers feel suspense even when they read a story for the second time (Dela-torre et al., 2018), which is unexpected if suspense is uncertainty; this is referred to as the paradox of suspense (Prieto-Pablos, 1998; Yanal, 1996). Considering Romeo and Juliet again, in the first view suspense is motivated by primarily by uncertainty over what will happen. Who will be hurt or killed in the fight? What will happen after marriage? However, at the beginning of the play we are told from forth the fatal loins of these two foes, a pair of star-crossed lovers take their life, and so the suspense is more about being invested in the plot than not knowing the outcome, aligning more with the second view: suspense can exist without uncertainty. We do not address the paradox of suspense directly in this paper, but we are guided by the debate to operationalise methods that encompass both views. The Hale model is closer to the traditional model of suspense as being about uncertainty. In contrast, the Ely model is more in line with the second view that uncertainty matters less than consequentially different outcomes. In NLP, suspense is studied most directly in natural language generation, with systems such as Dramatis (O'Neill and Riedl, 2014) and Suspenser (Cheong and Young, 2014), two planning-based story generators that use the theory of Gerrig and Bernardo (1994) that suspense is created when a protagonist faces obstacles that reduce successful outcomes. Our approach, in contrast, models suspense using general language models fine-tuned on stories, without planning and domain knowledge. The advantage is that the model can be trained on large volumes of available narrative text without requiring expensive annotations, making it more generalisable. Other work emphasises the role of characters and their development in story understanding (Bamman et al., 2014, 2013; Chaturvedi et al., 2017; Iyyer et al., 2016) or summarisation (Gorinski and Lap-ata, 2018). A further important element of narrative structure is plot, i.e., the sequence of events in which characters interact. Neural models have explicitly modelled events (Martin et al., 2018; Harrison et al., 2017; Rashkin et al., 2018) or the results of actions (Roemmele and Gordon, 2018; Liu et al., 2018a,b). On the other hand, some neural generation models (Fan et al., 2018) just use a hierarchical model on top of a language model; our architecture follows this approach. In order to formalise measures of suspense, we assume that a story consists of a sequence of sentences. These sentences are processed one by one, and the sentence at the current timepoint t is represented by an embedding e t (see Section 4 for how embeddings are computed). Each embedding is associated with a probability P ( e t ) . Continuations of the story are represented by a set of possible next sentences, whose embeddings are denoted by e it + 1 . The first measure of suspense we consider is surprise (Hale, 2001), which in the psycholinguistic literature has been successfully used to predict word-based processing effort (Demberg and Keller, 2008; Roark et al., 2009; Van Schijndel and Linzen, 2018a,b). Surprise is a backward-looking predictor: it measures how unexpected the current word is given the words that preceded it (i.e., the left context). Hale formalises surprise as the negative log of the conditional probability of the current word. For stories, we compute surprise over sentences. As our sentence embeddings e t include information about the left context e 1 ,..., e t  1 , we can write Hale surprise as: S Hale t =  log P ( e t ) (1) An alternative measure for predicting word-by-word processing effort used in psycholinguistics is entropy reduction (Hale, 2006). This measure is 1765 forward-looking: it captures how much the current word changes our expectations about the words we will encounter next (i.e., the right context). Again, we compute entropy at the story level, i.e., over sentences instead of over words. Given a probability distribution over possible next sentences P ( e it + 1 ) , we calculate the entropy of that distribution. Entropy reduction is the change of that entropy from one sentence to the next: H t =   i P ( e it + 1 ) log P ( e it + 1 ) U Hale t = H t  1  H t (2) Note that we follow Frank (2013) in computing entropy over surface strings, rather than over parse states as in Hale's original formulation. In the economics literature, Ely et al. (2015) have proposed two measures that are closely related to Hale surprise and entropy reduction. At the heart of their theory of suspense is the notion of belief in an end state. Games are a good example: the state of a tennis game changes with each point being played, making a win more or less likely. Ely et al. define surprise as the amount of change from the previous time step to the current time step. Intuitively, large state changes (e.g., one player suddenly comes close to winning) are more surprising than small ones. Representing the state at time t as e t , Ely surprise is defined as: S Ely t = ( e t  e t  1 ) 2 (3) Ely et al.'s approach can be adapted for modelling suspense in stories if we assume that each sentence in a story changes the state (the characters, places, events in a story, etc.). States e t then become sentence embeddings, rather than beliefs in end states, and Ely surprise is the distance between the current embedding e t and the previous embedding e t  1 . In this paper, we will use L1 and L2 distances; other authors (Li et al., 2018) experiment with information gain and KL divergence, but found worse performance when modelling suspense in games. Just like Hale surprise, Ely surprise models backward-looking prediction, but over representations, rather than over probabilities. Ely et al. also introduce a measure of forward-looking prediction, which they define as the expected difference between the current state e t and the next state e t + 1 : U Ely t = E [( e t  e it + 1 ) 2 ] =  i P ( e it + 1 )( e t  e it + 1 ) 2 (4) This is closely related to Hale entropy reduction, but again the entropy is computed over states (sen-tence embeddings in our case), rather than over probability distributions. Intuitively, this measure captures how much the uncertainty about the rest of the story is reduced by the current sentence. We refer to the forward-looking measures in Equations (2) and (4) as Hale and Ely uncertainty reduction, respectively. Ely et al. also suggest versions of their measures in which each state is weighted by a value  t , thus accounting for the fact that some states may be more inherently suspenseful than others: S  Ely t =  t ( e t  e t  1 ) 2 U  Ely t = E [  t + 1 ( e t  e it + 1 ) 2 ] (5) We stipulate that sentences with high emotional va-lence are more suspenseful, as emotional involvement heightens readers' experience of suspense. This can be captured in Ely et al.'s framework by assigning the  s the scores of a sentiment classifier. We now need to show how to compute the surprise and uncertainty reduction measures introduced in the previous section. This involves building a model that processes stories sentence by sentence, and assigns each sentence an embedding that encodes the sentence and its preceding context, as well as a probability. These outputs can then be used to compute a surprise value for the sentence. Furthermore, the model needs to be able to generate a set of possible next sentences (story contin-uations), each with an embedding and a probability. Generating upcoming sentences is potentially very computationally expensive since the number of continuations grows exponentially with the number of future time steps. As an alternative, we can therefore sample possible next sentences from a corpus and use the model to assign them embeddings and probabilities. Both of these approaches will produce sets of upcoming sentences, which we can then use to compute uncertainty reduction. While we have so far only talked about the next sentences, we will also experiment with uncertainty reduction computed using longer rollouts. Our overall approach leverages contextualised language models, which are a powerful tool in NLP when pretrained on large amounts of text and fine tuned on a specific task (Peters et al., 2018; Devlin et al., 2019). Specifically, we use Generative Pre-Training (GPT, Radford et al., 2018), a model which has proved successful in generation tasks (Radford et al., 2019; See et al., 2019). Hierarchical Model Previous work found that hierarchical models show strong performance in story generation (Fan et al., 2018) and understanding tasks (Cai et al., 2017). The language model and hierarchical encoders we use are unidirectional, which matches the incremental way in which human readers process stories when they experience suspense. Figure 1 depicts the architecture of our hierarchical model. 1 It builds a chain of representations that anticipates what will come next in a story, allowing us to infer measures of suspense. For a given sentence, we use GPT as our word encoder ( word enc in Figure 1) which turns each word in a sentence into a word embedding w i . Then, we use an RNN ( sent enc ) to turn the word embeddings of the sentences into a sentence embedding  i . Each sentence is represented by the hidden state of its last word, which is then fed into a second RNN 1 Model code and scripts for evaluation are available at https://github.com/dwlmt/Story-Untangling/ tree/acl-2020-dec-submission ( story enc ) that computes a story embedding. The overall story representation is the hidden state of its last sentence. Crucially, this model also gives us e t , a contextualised representation of the current sentence at point t in the story, to compute surprise and uncertainty reduction. Model training includes a generative loss (cid:96) gen to improve the quality of the sentences generated by the model. We concatenate the word representations w j for all word embeddings in the latest sentence with the latest story embedding e max ( t ) . This is run through affine ELU layers to produce enriched word embedding representations, analogous to the Deep Fusion model (Gulcehre et al., 2015), with story state instead of a translation model. The related Cold Fusion approach (Sriram et al., 2018) proved inferior. Loss Functions To obtain the discriminatory loss (cid:96) disc for a particular sentence s in a batch, we compute the dot product of all the story embeddings e in the batch, and then take the cross-entropy across the batch with the correct next sentence: (cid:96) disc ( e i = s t + 1 ) =  log exp ( e i = s t + 1  e t )  i exp ( e it + 1  e t ) (6) Modelled on Quick Thoughts (Logeswaran and Lee, 2018), this forces the model to maximise the dot product of the correct next sentence versus other sentences in the same story, and negative examples from other stories, and so encourages representations that anticipate what happens next. The generative loss in Equation (7) is a standard LM loss, where w j is the GPT word embeddings from the sentence and e max ( t ) is the story context that each word is concatenated with: (cid:96) gen =   j log P ( w j  w j  1 , w j  2 ,... ; e max ( t ) ) (7) The overall loss is (cid:96) disc + (cid:96) gen . More advanced generation losses (e.g., Zellers et al., 2019) could be used, but are an order of magnitude slower. We compute the measures of surprise and uncertainty reduction introduced in Section 3.1 using the output of the story encoder story enc . In addition to the contextualised sentence embeddings e t , this requires their probabilities P ( e t ) , and a distribution over alternative continuations P ( e i t + 1 ) . We implement a recursive beam search over a tree of future sentences in the story, looking between one and three sentences ahead (rollout). The 1767 probability is calculated using the same method as the discriminatory loss, but with the cosine similarity rather than the dot product of the embeddings e t and e it + 1 fed into a softmax function. We found that cosine outperformed dot product on inference as the resulting probability distribution over continuations is less concentrated. Dataset The overall goal of this work is to test whether the psycholinguistic and economic theories introduced in Section 3 are able to capture human intuition of suspense. For this, it is important to use actual stories which were written by authors with the aim of being engaging and interesting. Some of the story datasets used in NLP do not meet this criterion; for example ROC Cloze (Mostafazadeh et al., 2016) is not suitable because the stories are very short (five sentences), lack naturalness, and are written by crowdworkers to fulfill narrow objectives, rather than to elicit reader engagement and interest. A number of authors have also pointed out technical issues with such artificial corpora (Cai et al., 2017; Sharma et al., 2018). Instead, we use WritingPrompts (Fan et al., 2018), a corpus of circa 300k short stories from the /r/WritingPrompts subreddit. These stories were created as an exercise in creative writing, resulting in stories that are interesting, natural, and of suitable length. The original split of the data into 90% train, 5% development, and 5% test was used. Pre-processing steps are described in Appendix A. Annotation To evaluate the predictions of our model, we selected 100 stories each from the development and test sets of the WritingPrompts corpus, such that each story was between 25 and 75 sentence in length. Each sentence of these stories was judged for narrative suspense; five master workers from Amazon Mechanical Turk annotated each story after reading instructions and completing a training phase. They read one sentence at a time and provided a suspense judgement using the five-point scale consisting of Big Decrease in suspense (1% of the cases), Decrease (11%), Same (50%), Increase (31%), and Big Increase (7%). In contrast to prior work (Delatorre et al., 2018), a relative rather than absolute scale was used. Relative judgements are easier to make while reading, though in practice, the suspense curves generated are very similar, with a long upward trajectory and flattening or dip near the end. After finishing a story, annotators had GRU LSTM Loss 5.84 5.90 Discriminatory Acc. In the instructions, suspense was framed as dramatic tension , as pilot annotations showed that the term suspense was too closely associated with murder mystery and related genres. Annotators were asked to take the character's perspective when reading to achieve stronger inter-annotator agreement and align closely with literary notions of suspense. During training, all workers had to annotate a test story and achieve 85% accuracy before they could continue. Full instructions and the training story are in Appendix B. The inter-annotator agreement  (Krippendorff, 2011) was 0 . 52 and 0 . 57 for the development and test sets, respectively. Given the inherently subjective nature of the task, this is substantial agreement. This was achieved after screening out and replacing annotators who had low agreement for the stories they annotated (mean  < 0 . 35), showed suspiciously low reading times (mean RT < 600 ms per sentence), or whose story summaries indicated low-quality annotation. Training and Inference The training used SGD with Nesterov momentum (Sutskever et al., 2013) with a learning rate of 0 . 01 and a momentum of 0 . 9. Models were run with early stopping based on the mean of the accuracies of training tasks. For each batch, 50 sentence blocks from two different stories were chosen to ensure that the negative examples in the discriminatory loss include easy (other stories) and difficult (same story) sentences. We used the pretrained GPT weights but fine-tuned the encoder and decoder weights on our task. For the RNN components of our hierarchical model, we experimented with both GRU (Chung et al., 1768 2015) and LSTM (Hochreiter and Schmidhuber, 1997) variants. The GRU model had two layers in both sen enc and story enc ; the LSTM model had four layers each in sen enc and story enc . Both had two fusion layers and the size of the hidden layers for both model variants was 768. We give the results of both variants on the tasks of sentence generation and sentence discrimination in Table 1. Both perform similarly, with slightly worse loss for the LSTM variant, but faster training and better generation accuracy. Overall, model performance is strong: the LSTM variant picks out the correct sentence 54% of the time and generates it 46% of the time. This indicates that our architecture successfully captures the structure of stories. At inference time, we obtained a set of story continuations either by random sampling or by generation. Random sampling means that n sentences were selected from the corpus and used as continuations. For generation, sentences were generated using topk sampling (with k = 50) using the GPT language model and the approach of Radford et al. (2019), which generates better output than beam search (Holtzman et al., 2018) and can outperform a decoder (See et al., 2019). For generation, we used up to 300 words as context, enriched with the story sentence embeddings from the corresponding points in the story. For rollouts of one sentence, we generated 100 possibilities at each step; for rollouts of two, 50 possibilities and rollouts of three, 25 possibilities. This keeps what is an expensive inference process manageable. Importance We follow Ely et al. in evaluating weighted versions of their surprise and uncertainty reduction measure S  Ely t and U  Ely t (see Equation (5)). We obtain the  t values by taking the sentiment scores assigned by the VADER sentiment classifier (Hutto and Gilbert, 2014) to each sentence and multiplying them by 1 . 0 for positive sentiment and 2 . 0 for negative sentiment. The stronger negative weighting reflects the observation that negative consequences can be more important than positive ones (O'Neill, 2013; Kahneman and Tversky, 2013). Baselines We test a number of baselines as alternatives to surprise and uncertainty reduction derived from our hierarchical model. These baselines also reflect how much change occurs from one sentence to the next in a story: WordOverlap is the Jaccard similarity between the two sentences, GloveSim is the cosine similarity between the averaged Glove (Pennington et al., 2014) word embeddings of the two sentences, and GPTSim is the cosine similarity between the GPT embeddings of the two sentences. The  baseline is the weighted VADER sentiment score. Task The annotator judgements are relative (amount of decrease/increase in suspense from sentence to sentence), but the model predictions are absolute values. We could convert the model predictions into discrete categories, but this would fail to capture the overall arc of the story. Instead, we convert the relative judgements into absolute suspense values, where J t = j 1 +  + j t is the absolute value for sentence t and j 1 ,..., j t are the relative judgements for sentences 1 to t . We use  0 . 2 for Big Decrease,  0 . 1 for Decrease, 0 for Same, 0 . 1 for Increase, and 0 . 2 for Big Increase. 2 Both the absolute suspense judgements and the model predictions are normalised by converting them to z -scores. To compare model predictions and absolute suspense values, we use Spearman's  (Sen, 1968) and Kendall's  (Kendall, 1975). Rank correlation is preferred because we are interested in whether human annotators and models view the same part of the story as more or less suspenseful; also, rank correlation methods are good at detecting trends. We compute  and  between the model predictions and the judgements of each of the annotators (i.e., five times for five annotators), and then take the average. We then average these values again over the 100 stories in the test or development sets. As the human upper bound, we compute the mean pairwise correlation of the five annotators. Results Figure 2 shows surprise and uncertainty reduction measures and human suspense judgements for an example story (text and further examples in Appendix C). We performed model selection using the correlations on the development set, which are given in Table 2. We experimented with all the measures introduced in Section 3.1, computing sets of alternative sentences either us-2 These values were fitted with predictions (or cross-worker annotation) using 5-fold cross validation and an L1 loss to optimise the mapping. A constraint is placed so that Same is 0, increases are positive and decreases are negative with a minimum 0 . 05 distance between. ing generated continuations (Gen) or continuations sampled from the corpus (Cor), except for S Ely , which can be computed without alternatives. We compared the LSTM and GRU variants (see Section 4) and experimented with rollouts of up to three sentences. We tried L1 and L2 distance for the Ely measures, but only report L1, which always performed better. Discussion On the development set (see Table 2), we observe that all baselines perform poorly, indicating that distance between simple sentence representations or raw sentiment values do not model suspense. We find that Hale surprise S Hale performs well, reaching a maximum  of .675 on the development set. Hale uncertainty reduction U Hale , however, performs consistently poorly. Ely surprise S Ely also performs well, reaching as similar value as Hale surprise. Overall, Ely uncertainty reduction U Ely is the strongest performer, with  = . 698, numerically outperforming the human upper bound. Some other trends are clear from the development set: using GRUs reduces performance in all cases but one; rollout of more than one never leads to an improvement; sentiment weighting (prefix  in the table) always reduces performance, as it introduces considerable noise (see Figure 2). We therefore eliminate the models that correspond to these settings when we evaluate on the test set. For the test set results in Table 3 we also report upper and lower confidence bounds computed using the Fisher Z -transformation ( p < 0 . 05). On the test set, U Ely again is the best measure, with a correlation statistically indistinguishable from human performance (based on CIs). We find that absolute correlations are higher on the test set, presumably Prediction Model Roll     Human .553 .614 Baselines WordOverlap 1 .017 .026 GloveSim 1 .017 .029 GPTSim 1 .021 .031  1 .024 .036 S Hale -Gen GRU 1 .145 .182 LSTM 1 .434 .529 S Hale -Cor GRU 1 .177 .214 LSTM 1 .580 .675 U Hale -Gen GRU 1 .036 .055 LSTM 1 .009 .016 U Hale -Cor GRU 1 .048 .050 LSTM 1 .066 .094 S Ely GRU 1 .484 .607 LSTM 1 .427 .539 S  Ely GRU 1 .089 .123 LSTM 1 .115 .156 U Ely -Gen GRU 1 .241 .161 2 .304 .399 LSTM 1 .610 .698 2 .393 .494 U Ely -Cor GRU 1 .229 .264 2 .512 .625 3 .515 .606 LSTM 1 .594 .678 2 .564 .651 3 .555 .645 U  Ely -Gen GRU 1 .216 .124 2 .219 .216 LSTM 1 .474 .604 2 .316 .418 U  Ely -Cor GRU 1 .205 .254 2 .365 .470 LSTM 1 .535 .642 2 .425 .534 Table 2: Development set results for WritingPrompts for generated (Gen) or corpus sampled (Cor) alternative continuations;  indicates sentiment weighting. judgements on the WritingPrompts dataset. The overall best predictor is U Ely , uncertainty reduction computed over story representations. This measure combines the probability of continuation ( S Hale ) with distance between story embeddings ( S Ely ), which are both good predictors in their own right. This finding supports the theoretical claim that suspense is an expectation over the change in future states of a game or a story, as advanced by Ely et al. (2015). Task and Dataset An interesting question is whether the peaks in suspense in a story correspond to important narrative events. Such events are sometimes called turning points (TPs) and occur at certain positions in a movie according to screenwriting theory (Cutting, 2016). A corpus of movie synopses annotated with turning points is available in the form of the TRIPOD dataset (Papalampidi et al., 2019). We can therefore test if surprise or uncertainty reduction predict TPs in TRIPOD. As our model is trained on a corpus of short stories, this will also serve as an out-of-domain evaluation. Papalampidi et al. (2019) assume five TPs: 1. Opportunity, 2. Change of Plans, 3. Point of no Return, 4. Major Setback, and 5. Climax. They derive a prior distribution of TP positions from their test set, and use this to constrain predicted turning points to windows around these prior positions. We follow this approach and select as the predicted TP the sentence with the highest surprise or uncertainty reduction value within a given constrained window. We report the same baselines as in the previous experiment, as well as the Theory Baseline, Dev D  Test D  Human Not reported 4.30 (3.43) Theory Baseline 9.65 (0.94) 7.47 (3.42) TAM 7.11 (1.71) 6.80 (2.63) WordOverlap 13.9 (1.45) 12.7 (3.13) GloveSim 10.2 (0.74) 10.4 (2.54) GPTSim 16.8 (1.47) 18.1 (4.71)  11.3 (1.24) 11.2 (2.67) S Hale -Gen 8.27 (0.68) 8.72 (2.27) U Hale -Gen 10.9 (1.02) 10.69 (3.66) S Ely 9.54 (0.56) 9.01 (1.92) S  Ely 9.95 (0.78) 9.54 (2.76) U Ely -Gen 8.75 (0.76) 8.38 (1.53) U Ely -Cor 8.74 (0.76) 8.50 (1.69) U  Ely -Gen 8.80 (0.61) 7.84 (3.34) U  Ely -Cor 8.61 (0.68) 7.78 (1.61) Table 4: TP prediction on the TRIPOD development and test sets. which uses screenwriting theory to predict where in a movie a given TP should occur (e.g., Point of No Return theoretically occurs 50% through the movie). This baseline is hard to beat (Papalampidi et al., 2019). Results and Discussion Figure 3 plots both gold standard and predicted TPs for a sample movie synopsis (text and further examples in Appendix D). The results on the TRIPOD development and test sets are reported in Table 4 (we report both due to the small number of synopses in TRIPOD). We use our best LSTM model with a of rollout of one; the distance measure for Ely surprise and uncertainty reduction is now L2 distance, as it outperformed L1 on TRIPOD. We report results in terms of D , the normalised distance between gold standard and predicted TP positions. On the test set, the best performing model with D = 7 . 78 is U  Ely -Cor, with U  Ely -Gen only slightly worse. It is outperformed by TAM, the best model of Papalampidi et al. (2019), which however requires TP annotation at training time. U  Ely -Cor is close to the Theory Baseline on the test set, an impressive result given that our model has no TP supervision and is trained on a different domain. The fact that models with sentiment 1771 0 10 20 30 40 50 0 1 2 3 4 5 6 Sentence S u s p e n s e Figure 3: Movie 15 Minutes, S Hale , S Ely , U Ely , U  Ely ,  theory baseline,  TP annotations, triangles are predicted TPs. Our overall findings suggest that by implementing concepts from psycholinguistic and economic theory, we can predict human judgements of suspense in storytelling. That uncertainty reduction ( U Ely ) outperforms probability-only ( S Hale ) and state-only ( S Ely ) surprise suggests that, while consequential state change is of primary importance for suspense, the probability distribution over the states is also a necessary factor. Uncertainty reduction therefore captures the view of suspense as reducing paths to a desired outcome, with more consequential shifts as the story progresses (O'Neill and Riedl, 2014; Ely et al., 2015; Perreault, 2018). This is more in line with the Smuts (2008) Desire-Frustration view of suspense, where uncertainty is secondary. Strong psycholinguistic claims about suspense are difficult to make due to several weaknesses in our approach, which highlight directions for future research: the proposed model does not have a higher-level understanding of event structure; most likely it picks up the textual cues that accompany dramatic changes in the text. One strand of further work is therefore analysis: Text could be artificially manipulated using structural changes, for example by switching the order of sentences, mixing multiple stories, including a summary at the beginning that foreshadows the work, masking key suspenseful words, or paraphrasing. An analogue of this would be adversarial examples used in computer vision. Additional annotations, such as how certain readers are about the outcome of the story, may also be helpful in better understanding the relationship between suspense and uncertainty. Automated interpretability methods as proposed by Sundarara-jan et al. (2017), could shed further light on models' predictions. The recent success of language models in wide-ranging NLP tasks (e.g., Radford et al., 2019) has shown that language models are capable of learning semantically rich information implicitly. However, generating plausible future continuations is an essential part of the model. In text generation, Fan et al. (2019) have found that explicitly incorporating coreference and structured event representations into generation produces more coherent generated text. A more sophisticated model would incorporate similar ideas. Autoregressive models that generate step by step alternatives for future continuations are computationally impractical for longer rollouts and are not cognitively plausible. They also differ from the Ely et al. (2015) conception of suspense, which is in terms of Bayesian beliefs over a longer-term future state, not step by step. There is much recent work (e.g., Ha and Schmidhuber (2018); Gregor et al. (2019)), on state-space approaches that model beliefs as latent states using variational methods. In principle, these would avoid the brute-force calculation of a rollout and conceptually, anticipating longer-term states aligns with theories of suspense. This paper is a baseline that demonstrates how modern neural network models can implicitly represent text meaning and be useful in a narrative context without recourse to supervision. It provides a springboard to further interesting applications and research on suspense in storytelling. The authors would like to thank the anonymous reviewers, Pinelopi Papalampidi and David Hodges for reviews of the annotation task, the AMT annotators, and Mirella Lapata, Ida Szubert, and Elizabeth Nielson for comments on the paper. Wilmot's work is funded by an EPSRC doctoral training award.
abstain abstain method abstain objective result result abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain method result abstain result other other other other other method abstain other other other objective other other other other method method abstain abstain abstain abstain abstain abstain method method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain other abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain method abstain method method method other abstain method abstain method abstain abstain other other abstain method abstain method abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result result result abstain abstain abstain result abstain method abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain other other