We explore the challenge of action prediction from textual descriptions of scenes, a testbed to approximate whether text inference can be used to predict upcoming actions. As a case of study, we consider the world of the Harry Potter fantasy novels and inferring what spell will be cast next given a fragment of a story. Spells act as keywords that abstract actions (e.g. Alohomora' to open a door) and denote a response to the environment. This idea is used to automatically build HPAC , a corpus containing 82 836 samples and 85 actions. We then evaluate different baselines. Among the tested models, an LSTM -based approach obtains the best performance for frequent actions and large scene descriptions, but approaches such as logistic regression behave well on infrequent actions. Natural language processing ( NLP ) has achieved significant advances in reading comprehension tasks (Chen et al., 2016; Salant and Berant, 2017). These are partially due to embedding methods (Mikolov et al., 2013; Devlin et al., 2018) and neural networks (Rosenblatt, 1958; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017), but also to the availability of new resources and challenges. For instance, in cloze-form tasks (Hermann et al., 2015; Bajgar et al., 2016), the goal is to predict the missing word given a short context. Weston et al. (2015) presented baBI, a set of proxy tasks for reading comprenhension. In the SQuAD corpus (Rajpurkar et al., 2016), the aim is to answer questions given a Wikipedia passage. Kocisky et al. (2018) introduce NarrativeQA, where answering the questions requires to process entire stories. In a related line, Frermann et al. (2017) use fictional crime scene investigation data, from the CSI series, to define a task where the models try to answer the question: who committed the crime?'. In an alternative line of work, script induction (Schank and Abelson, 1977) has been also a useful approach to evaluate inference and semantic capabilities of NLP systems. Here, a model processes a document to infer new sequences that re-flect events that are statistically probable (e.g. go to a restaurant, be seated, check the menu, . . . ). For example, Chambers and Jurafsky (2008) introduce narrative event chains, a representation of structured knowledge of a set of events occurring around a protagonist. They then propose a method to learn statistical scripts, and also introduce two different evaluation strategies. With a related aim, Pichotta and Mooney (2014) propose a multi-event representation of statistical scripts to be able to consider multiple entities. These same authors (Pichotta and Mooney, 2016) have also studied the abilities of recurrent neural networks for learning scripts, generating upcoming events given a raw sequence of tokens, using BLEU (Pap-ineni et al., 2002) for evaluation. This paper explores instead a new task: action prediction from natural language descriptions of scenes. The challenge is addressed as follows: given a natural language input sequence describing the scene, such as a piece of a story coming from a transcript, the goal is to infer which action is most likely to happen next. Contribution We introduce a fictional-domain English corpus set in the world of Harry Potter novels. The domain is motivated by the existence of a variety of spells in these literary books, associated with keywords that can be seen as unambiguous markers for actions that potentially relate to the previous context. This is used to automatically create a natural language corpus coming from hundreds of users, with different styles, interests and writing skills. We then train a number of standard baselines to predict upcoming actions, a task that requires to be aware of the context. In particular, we test a number of generic models, from a simple logistic regression to neural models. Experiments shed some light about their strengths and weaknesses and how these are related to the frequency of each action, the existence of other semantically related actions and the length of the input story. To build an action prediction corpus, we need to: (1) consider the set of actions, and (2) collect data where these occur. Data should come from different users, to approximate a real natural language task. Also, it needs to be annotated, determining that a piece of text ends up triggering an action. These tasks are however time consuming, as they require annotators to read vast amounts of large texts. In this context, machine comprehension resources usually establish a compromise between their complexity and the costs of building them (Hermann et al., 2015; Kocisky et al., 2018). We rely on an intuitive idea that uses transcripts from the Harry Potter world to build up a corpus for textual action prediction. The domain has a set of desirable properties to evaluate reading comprehension systems, which we now review. Harry Potter novels define a variety of spells. These are keywords cast by witches and wizards to achieve purposes, such as turning on a light (Lu-mos'), unlocking a door (Alohomora') or killing (Avada Kedavra'). They abstract complex and non-ambiguous actions. Their use also makes it possible to build an automatic and self-annotated corpus for action prediction. The moment a spell occurs in a text represents a response to the environment, and hence, it can be used to label the preceding text fragment as a scene description that ends up triggering that action. Table 1 illustrates it with some examples from the original books. This makes it possible to consider texts from the magic world of Harry Potter as the domain for the action prediction corpus, and the spells as the set of eligible actions. 1 Determining the length of the preceding context, namely snippet , that has to be 1 Note that the corpus is built in an automatic way and some occurrences might not correspond to actions, but for example, to a description of the spell or even some false positive samples. Related to this, we have not censored the content of the stories, so some of them might contain adult content. considered as the scene description is however not trivial. This paper considers experiments (  4) using snippets with the 32, 64, 96 and 128 previous tokens to an action. We provide the needed scripts to rebuild the corpus using arbitrary lengths. 2 2.2 Data crawling The number of occurrences of spells in the original Harry Potter books is small (432 occurrences), which makes it difficult to train and test a machine learning model. However, the amount of available fan fiction for this saga allows to create a large corpus. For HPAC , we used fan fiction (and only fan fiction texts) from https://www. fanfiction.net/book/Harry-Potter/ and a version of the crawler by Milli and Bamman (2016). 3 We collected Harry Potter stories written in English and marked with the status com-pleted'. From these we extracted a total of 82 836 spell occurrences, that we used to obtain the scene descriptions. Table 2 details the statistics of the corpus (see also Appendix A). Note that similar to Twitter corpora, fan fiction stories can be deleted over time by users or admins, causing losses in the dataset. 4 Preprocessing We tokenized the samples with (Manning et al., 2014) and merged the occurrences of multi-word spells into a single token. This work addresses the task as a classification problem, and in particular as a sequence to label classification problem. For this reason, we rely on standard models used for this type of task: multinomial logistic regression, a multi-layered perceptron, convolutional neural networks and long short-term memory networks. We outline the essentials of each of these models, but will treat them as black boxes. In a related line, Kaushik and Lipton (2018) discuss the need of providing rigorous baselines that help better understand the improvement coming from future and complex models, and also the need of not demanding architectural novelty when introducing new datasets. https://github.com/aghie/hpac 3 Due to the website's Terms of Service, the corpus cannot be directly released. 4 They also can be modified, making it unfeasible to retrieve some of the samples. Text fragment Action Ducking under Peeves, they ran for their lives, right to the end of the corridor where they slammed into a door and it was locked. This is it!' Ron moaned, as they pushed helplessly at the door, We're done for! This is the end!' They could hear footsteps, Filch running as fast as he could toward Peeves's shouts. Oh, move over', Hermione snarled. She grabbed Harry's wand, tapped the lock, and whispered,  Alohomora '. Unlock the door And then, without warning, Harry's scar exploded with pain. It was agony such as he had never felt in all his life; his wand slipped from his fingers as he put his hands over his face; his knees buckled; he was on the ground and he could see nothing at all; his head was about to split open. From far away, above his head, he heard a high, cold voice say, Kill the spare.' A swishing noise and a second voice, which screeched the words to the night:  Avada Kedavra ' Kill a target Harry felt himself being pushed hither and thither by people whose faces he could not see. Then he heard Ron yell with pain. What happened?' said Hermione anxiously, stopping so abruptly that Harry walked into her. Ron, where are you? Oh, this is stupid'  Lumos ' Turn on a light Table 1: Examples from the Harry Potter books showing how spells map to reactions to the environment. special case of language modelling, where the output vocabulary is restricted to the size of the ac-tion' vocabulary. Also, note that the performance for this task is not expected to achieve a perfect accuracy, as there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty. The source code for the models can be found in the GitHub repository mentioned above. Notation w 1: n denotes a sequence of words w 1 , ..., w n that represents the scene, with w i  V . F  (  ) is a function parametrized by  . The task is cast as F : V n  A , where A is the set of actions. The input sentence w 1: n is encoded as a one-hot vector, v (total occurrence weighting scheme). Multinomial Logistic Regression Let MLR  ( v ) be an abstraction of a multinomial logistic regression parametrized by  , the output for an input v is computed as the arg max a  AP ( y = a | v ) , where P ( y = a | v ) is a softmax function, i.e, P ( y = a | v ) = e Wa  v (cid:80) Aa (cid:48) e Wa (cid:48) v . MultiLayer Perceptron We use one hidden layer with a rectifier activation function ( relu ( x ) = max (0 , x ) ). The output is computed as MLP  ( v ) = softmax ( W 2  relu ( W  v + b )+ b 2 ) . The input sequence is represented as a sequence of word embeddings, w 1: n , where w i is a concatenation of an internal embedding learned during the training process for the word w i , and a pre-trained embedding extracted from GloVe (Pen-nington et al., 2014) 5 , that is further fine-tuned. Long short-term memory network (Hochre-iter and Schmidhuber, 1997): The output for an element w i also depends on the output of w i  1 . The LSTM  ( w 1: n ) 6 takes as input a sequence of word embeddings and produces a sequence of hidden outputs, h 1: n ( h i size set to 128). The last output of the LSTM  , h n , is fed to a MLP  . Convolutional Neural Network (LeCun et al., 1995; Kim, 2014). It captures local properties over continuous slices of text by applying a convolution layer made of different filters. We use a wide convolution, with a window slice size of length 3 and 250 different filters. The convolutional layer uses a relu as the activation function. The output is fed to a max pooling layer, whose output vector is passed again as input to a MLP  . Setup All MLP  's have 128 input neurons and 1 hidden layer. We trained up to 15 epochs using mini-batches (size=16), Adam (lr= 0 . 001 ) (Kingma and Ba, 2015) and early stopping. Table 3 shows the macro and weighted F-scores for the models considering different snippet sizes. 7 5 http://nlp.stanford.edu/data/glove. 6 n is set to be equal to the length of the snippet. 7 As we have addressed the task as a classification problem, we will use precision, recall and F-score as the evaluation metrics. To diminish the impact of random seeds and local minima in neural networks, results are averaged across 5 runs. 8 Base' is a majority-class model that maps everything to Avada Kedavra', the most common action in the training set. This helps test whether the models predict above chance performance. When using short snippets (size=32), disparate models such as our MLR , MLP and LSTM s achieve a similar performance. As the snippet size is increased, the LSTM -based approach shows a clear improvement on the weighted scores 9 , something that happens only marginally for the rest. However, from Table 3 it is hard to find out what the approaches are actually learning to predict. To shed some light, Table 4 shows their performance according to a ranking metric, recall at k . The results show that the LSTM -based approach is the top performing model, but the MLP obtains just slightly worse results. Recall at 1 is in both cases low, which suggests that the task is indeed complex and that using just LSTM s is not enough. It is also possible to observe that even if the models have difficulties to correctly predict the action as a first option, they develop certain sense of the scene and consider the right one among their top choices. Table 5 delves into this by splitting the performance of the model into infrequent and frequent actions (above the average, i.e. those that occur more than 98 times in the training set, a total of 20 actions). There is a clear gap between 8 Some macro F-scores do not lie within the Precision and Recall due to this issue. 9 For each label, we compute their average, weighted by the number of true instances for each label. The F-score might be not between precision and recall. the performance on these two groups of actions, with a  50 points difference in recall at 5 . Also, a simple logistic regression performs similar to the LSTM on the infrequent actions. Error analysis 10 Some of the misclassifications made by the LSTM approach were semantically related actions and counter-actions. For example, Colloportus' (to close a door) was never predicted. The most common mis-classification (14 out of 41) was Alohomora' (to unlock a door), which was 5 times more frequent in the training corpus. Similarly, Nox' (to extinguish the light from a wand) was correctly predicted 6 times, meanwhile 36 mis-classifications corre-10 Made over one of the runs from the LSTM -based approach and setting the snippet size to 128 tokens. spond to Lumos' (to light a place using a wand), which was 6 times more frequent in the training set. Other less frequent spells that denote vision and guidance actions, such as Point me' (the wand acts a a compass pointing North) and Homenum revelio' (to revel a human presence) were also mainly misclassified as Lumos'. This is an indicator that the LSTM approach has difficulties to disambiguate among semantically related actions, especially if their occurrence was unbalanced in the training set. This issue is in line with the tendency observed for recall at k . Spells intended for much more specific purposes, according to the books, obtained a performance significantly higher than the average, e.g. F-score(Riddikulus')=63.54, F-score(Expecto Pa-tronum')=55.49 and F-score(Obliviate')=47.45. As said before, the model is significantly biased towards frequent actions. For 79 out of 84 gold actions in the test set, we found that the samples tagged with such actions were mainly classified into one of the top 20 most frequent actions. Human comparison We collected human annotations from 208 scenes involving frequent actions. The accuracy/F-macro/F-weighted was 39.20/30.00/40.90. The LSTM approach obtained 41.26/25.37/39.86. Overall, the LSTM approach obtained a similar performance, but the lower macro F-score by the LSTM could be an indicator that humans can distinguish within a wider spectrum of actions. As a side note, super-human performance it is not strange in other NLP tasks, such as sentiment analysis (Pang et al., 2002). We explored action prediction from written stories. We first introduced a corpus set in the world of Harry Potter's literature. Spells in these novels act as keywords that abstract actions. This idea was used to label a collection of fan fiction. We then evaluated standard NLP approaches, from logistic regression to sequential models such as LSTM s. The latter performed better in general, although vanilla models achieved a higher performance for actions that occurred a few times in the training set. An analysis over the output of the LSTM approach also revealed difficulties to discriminate among semantically related actions. The challenge here proposed corresponded to a fictional domain. A future line of work we are interested in is to test whether the knowledge learned with this dataset could be transferred to real-word actions (i.e. real-domain setups), or if such transfer is not possible and a model needs to be trained from scratch. This work has received support from the TELEPARES-UDC project (FFI2014-51978-C2-2-R) and the ANSWER-ASAP project (TIN2017-85160-C2-1-R) from MINECO, and from Xunta de Galicia (ED431B 2017/01), and from the European Research Council (ERC), under the European Union's Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150).
objective objective abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain objective objective abstain abstain method abstain abstain abstain method other