We present a complete pipeline to extract characters in a novel and link them to their direct-speech utterances. Our model is divided into three independent components: extracting direct-speech, compiling a list of characters, and attributing those characters to their utterances. Although we find that existing systems can perform the first two tasks accurately, attributing characters to direct speech is a challenging problem due to the narrator's lack of explicit character mentions, and the frequent use of nominal and pronominal coreference when such explicit mentions are made. We adapt the progress made on Dialogue State Tracking to tackle a new problem: attributing speakers to dialogues. This is the first application of deep learning to speaker attribution, and it shows that is possible to overcome the need for the hand-crafted features and rules used in the past. Our full pipeline improves the performance of state-of-the-art models by a relative 50% in F1-score. Natural Language Processing has enabled a quantitative improvement in the humanities, by allowing for large-scale statistical measurements to be taken over hundreds of thousands of books compared to the order of tenths a human could analyse in a much longer time span. Some examples of large-scale literary analyses include studies on characters and their descriptions within the novel, mostly focused on gender differences (Underwood et al., 2018; Kraicer and Piper, 2019), and studies on character's relations by extracting social networks from novels (Labatut and Bost, 2019; Jayannavar et al., 2015). Most of these studies demand special attention to dialogues, being a major part of character expression and interaction with other characters. Dialogues play an instrumental role in plot developWork done during internship at Amazon ment, frequently encompassing focal plot moments, especially in fiction  which is also the focus for this study. Here we aim to identify direct-speech utterances that form part of dialogues and associate them with the speaking characters. Such information is not only useful to enable large-scale socio-temporal studies but also crucial to many downstream challenging tasks like narrative understanding (Iyyer et al., 2016) and summarising (Ladhak et al., 2020). Further, the high-quality dialogue-character association is pertinent for generating engaging text-to-speech for novels with distinct voice profiles for characters. In the past, models that link direct speech to characters have been dominated by predefined rules (Muzny et al., 2017) or hand-crafted features (He et al., 2013). When evaluating these models the authors also presumed that a character list, together with the character's aliases, has been precompiled and that direct-speech text has been extracted. Although extensions to these models that extract speaking characters in a fully automated manner exist, it is unclear what impact does the automation of the aforementioned steps has on the final performance of the model. Moreover, the models have only been tested against a small dataset of three books from the same time period. These are the two questions that we aim to answer in this paper: i) how can we build flexible models that can generalise and improve with increasing dataset sizes?, and ii) what is the impact of errors propagating from each component of the pipeline, and thus where should we focus future efforts? To answer these questions, we focus on building deep learning models with the necessary inductive biases and flexibility to learn nuanced features when given a large enough dataset, as opposed to hand-crafted rules that need revisiting to generalise to different time periods, writing styles, genres, or even languages. Moreover, we present a separate evaluation of pipeline's each component. Attributing speakers to direct speech is a common problem for two related domains: news and literature. However, previous work (O'Keefe et al., 2012) has shown that models do not generalise to both domains. Their best model obtained an accuracy of 92 . 4% and 84 . 1% on their two newswire datasets, whilst they only found a 53 . 3% accuracy when evaluating the same model on a literature dataset. We therefore focus on summarising progress in quote attribution for literary fiction. The early models targeting literary texts (Glass and Bangay, 2007) were based on the identification of speech verbs and their actors. However, the proportion of dialogues accompanied by a speech-verb and an explicit mention to a character can be as low as 20% of the total quotes for some books. For this reason, consequent work (O'Keefe et al., 2012; Elson and McKeown, 2010) shifted their focus to attributing speakers to dialogues where the character is not explicitly mentioned, incorporating rules to exploit the sequential nature of conversations. These models could not improve the results of a simple Nearest Mention (NM) baseline that obtained a 53 . 3% accuracy on their test set. Finally, current state-of-the-art models (Muzny et al., 2017; He et al., 2013) demonstrated how the simple nearest mention baseline could be beaten through a combination of rules and learning. Both models present analysis on a limited setup: i) they report performance on a test set comprised of two books  Jane Austen's Emma , and Anton Chekhov's The Steppe ,  and have therefore not been bench-marked on a wider range of styles or time periods, and ii) they assume the ideal circumstance of a pre-compiled list of characters, with character aliases and genders provided. We relax the second assumption when we evaluate our model, to estimate the end-to-end performance on a more diverse dataset of fifteen books. The task of speaker attribution is also closely related to other dialogue sequence problems. One such umbrella technique to solve these problems is Dialogue State Tracking or simply DST, where a system is tasked with estimating some conversation state variables usually the user's goals and intents. We are first to apply DST for the purpose of speaker attribution. Our proposed DST-based formulation requires modification to the utterance encoder with focus on non-dialogue context, and state-variable that can generalise to states not seen in the training set. We adapt a BERT-based DST model (Lai et al., 2020) to track the speaker for every single utterance instead of tracking the user's goals and intents. Our work follows a similar line of thought to Ren et al. (2018); Lai et al. (2020), where the model is given a list of candidate intents (speakers in our case) embedded as inputs to the problem so that the model can generalise to unseen goals (speak-ers) during test time. The task of our model is to generate a score for each utterance against every candidate speaker. To recap, we present an end-to-end pipeline for speaker to dialogue attribution that leverages re-cent advances in large pretrained Language Models casting the problem as a Dialogue State Tracking. We empirically show that our model is capable of generalising to different styles more reliably as compared to prior hand-crafted features-based systems. Further, we present this comparative study on literary texts ranging from the 1900s to the 2010s, which are more varied and diverse compared to past studies. Note that usually such dataset are effort-intensive to create and not publicly available due to lack of rights to redistribution, which makes the reported result very interesting for the wider community. Our annotations consist of two independent layers, one focusing on direct speech, and one focusing on clustering mentions that refer to the same character entity. Example 1: Excerpt from 2001: A Space Odyssey. Annotated direct speech is in bold, and the annotated attributed character entity inside a blue box. Poole was asleep, and Bowman was reading on the control deck, when Hal announced: Er Dave , I have a report for you. HAL What's up? DAVID BOWMAN We have another bad AE-35 unit. My fault predictor indicates failure within twenty-four hours. HAL For the first layer, the annotator selects the span of text representing a character's direct speech. It is usually found within quotation marks, but this is not a necessary or sufficient requirement. The annotator then attributes a character entity to the utterance. Example 1 presents a typical conversation with instances of coreference (Dave and Bow-5821 man both refer to David Bowman), and implicit attribution (third and fourth paragraphs) where no character is explicitly mentioned by the narrator. The second layer of annotations focuses on characters and their mentions. We follow Bamman et al. (2014) and distinguish character mentions (e.g. Dave, David, Bowman, Dr. Bowman) in the text and character entities (e.g. DAVID BOW-MAN), to which mentions refer to. See the text in italics within Example 1 to find some of the annotated character mentions. Note that we don't include pronominal mentions. These mentions are then clustered per book into character entities by the annotators. We annotate a collection of 15 books sampled from the most popular titles from time epochs 1881 2018 . The annotation is carried out by 3 expert English native annotators, each reading the book in sequential order, over a BRAT 1 based annotation interface. In case the annotation from any single annotator is different, a master annotator goes through the cases and makes the correction, resulting in a dataset with a very high agreement (Cohen's Kappa greater than 0.9). Across the books, the number of annotated dialogues varies from 200 to 5000 and characters from less than 10 to 200. We would refer these books by IDs 1 through 15 and the existing 3 books as E1 ( Emma ), E2 ( Pride and Prejudice ) and E3 ( The Steppe ). We divide the model into three main tasks: identifying quotes, extracting unique characters and their aliases, and attributing dialogues to the extracted characters. Our goal is to reduce the amount of hand-crafted rules (usually heavily biased to the small subset of documents of prior studies) where performance can be improved, and allow the model to learn nuanced features that allow it to generalise better when given a large enough dataset. We first introduce our direct speech identification component, which is purely rule-based due to the simplicity of the problem and since improving this aspect is not part of our core contribution. Afterward, we focus on identifying characters and compare NER and coreference resolution deep learning models to simple rule-based systems. Finally, we discuss the focal aspect of our contribution a DST architectural adaptation to solve the speaker attribution of quotes. Direct speech in fiction is usually denoted with quotation marks, although there are exceptions such as Ali Smith's Summer , where speech marks are completely removed and dialogues blend in with the rest of the text, or Joyce's Ulysses that introduces speech with dashes. Here we ignore such instances, which are not present in our dataset, and focus on the most common case where direct speech is marked by quotation marks. Further, we find that for English over  95 % of the dialogues (as analysed over a large collection of popular books) follow open-close quotation-pair variation. See Steinbach et al. (2011) for an in-depth review of the topic. In the case of extracting quotation marks, simple rules can achieve almost perfect performance. As in O'Keefe et al. (2012), we use a regular expression to detect opening and closing quotation marks that denote the presence of direct speech. Although characters are central to most literary analyses, identifying them automatically from a novel remains an unsolved problem. We split the character identification task into: i) identifying mentions in the text that refer to characters, and ii) clustering those mentions into unique character entities. Similar to direct speech identification, we do not focus on improving the architecture for character identification. As both of these form input to our core DST module, we re-purpose the best of existing techniques. However, unlike previous studies, we do analyse and report the impact of these components on end-to-end performance to guide future research. Extracting entities from text is normally done over short documents, such as Wikipedia pages. But literature brings unique challenges to the field: novels tend to be long documents, demanding effi-cient algorithms, and requiring models to be able to link far apart mentions. We present an evaluation of Named Entity Recognition (NER) to detect mentions, together with the effectiveness of coreference resolution to cluster character mentions into entities. We find that although NER achieves a similar performance to a simple rule-based system, coreference resolu-tion's performance on clustering characters is significantly poorer than a simple rule-based character clustering technique, and future work should fo-5822 Figure 1: Diagram of our speaker attribution pipeline. We present a comparison between an out-of-domain NER model, trained on the CoNLL-2003 dataset (Tjong Kim Sang and De Meulder, 2003) with a simple rule-based baseline that focuses on identifying all the characters that speak explicitly. It finds the subject of the narrator's explicit dialogue attribution signals (defined by the 40 most frequent speech denoting verbs such as said, answered, ... ). In Table 1, we show a summary of the aliases variations found in our dataset and their frequency. Since our core contribution is not to improve character clustering, and our dataset of character aliases is small, we do not develop a custom model for it and merely compare two different clustering tech-Type Freq Example ([mentions]  entity) Full Name Variation 67% [Harry, Potter, Mr. Potter, Harry Potter]  HARRY POTTER) Dimunitives 7% [Lizzy, Eliza]  ELIZABETH Professional 15 . niques: i) an out-of-domain coreference resolution system, and ii) a simple set of rules that cluster characters according to their names. In both cases, we build a graph where nodes are character mentions, and edges are attached to all two compatible nodes. See the top right box on Clustering Mentions in Figure 1 for an example of such graph. On one hand, in the case of coreference resolution, two nodes are compatible if they appear together at least in two coreferent clusters. Clusters of characters are formed by finding all disconnected subgraphs within the graph. On the other hand, although character aliases can be of any kind and might not be related to each other by name, most of the ones appearing in literature are variations of the character's full name (See Table 1). We build a rule-based algorithm that deems two names incompatible if, i) The first names of the characters are different (the first names do not match exactly) or the shorter first name is not exactly inside the longest one. This also takes care of a few diminutive forms (like Eliza for Elizabeth). We split the graph into disconnected subgraphs of compatible names, and find those nodes that are ambiguous, i.e., nodes whose first neighbour connections contain more than one title, first name, or surname. Removing those nodes we can form unambiguous clusters of characters that share the same title, name, and surname. As opposed to previous work (Bamman et al., 2014; Elson et al., 2010), where ambiguous names would be merged to the closest entity mentioned in the text, we allow ambiguous nodes to either form their own cluster, or be part of any of their first neighbour nodes' clusters. We use the text to resolve this ambiguity by finding the most mentioned cluster among the possible clusters in the 20 paragraph vicinity of the ambiguous mention. In this way, we can retain characters such as Mrs. Bennet in Pride and Prejudice , without merging them to other members of the Bennet family, since they are prominent enough to be given their own cluster. In this section, we present an adaptation of Dialogue State Tracking to speaker attribution. In DST, it is a challenge to produce models that can work with dynamic ontologies and unseen slot values such that the user can request information on any slot (movies, restaurants, ...) and use any value (the type of food, the price, ...) that has not necessarily been seen at test time. In the same way, we can't simply use a general fixed tag set of characters beyond the level of an individual novel, since we want our model to generalise to unseen novels and unseen characters during test time. We will therefore embed the character's mentions within the inputs of the model as done in state-of-the-art DST (Lai et al., 2020). Below, we discuss in detail how we adapt DST to model speaker attribution in novels. Although our dataset is annotated at the level of word spans, the odds are high that disconnected spans in the same paragraph are attributed to the same speaker. We find that this rule is violated on less than 5% of the paragraphs that contain more than one disconnected span. Therefore, as in He et al. (2013), our model will be trained on attributing speakers at the level of paragraphs. Regarding the model inputs, we split the text into conversations. Denoting paragraphs with no direct speech as narratives, we segment conversations by restricting the number of intervening narratives between direct speech utterances to one. If more than one narrative separates two direct speech utterances, the conversation is split into two different conversations. Given a set of n utterances that define a conversation, u = { u 0 , u 1 , ..., u n  1 } , a set of l mentions to candidate characters, m = { m 0 , m 1 , ..., m l  1 } , and a set of k candidate characters entities linked to those mentions, c = { c 0 , c 1 , ..., c k  1 } , where k < = l , we wish to model the probability for each candidate character entity, c i , being the speaker of a given utterance, u j . This probability will be denoted as P ( c i | u j , u ) . Let's denote as  the embedding model that transforms word tokens in vectors (equivalently IRD (cid:15) space), where D (cid:15) is the output dimension of the embedding model. In this work, we chose a Distil-BERT model (Sanh et al., 2019) for  . As in Figure 1, we generate an embedding for every utterance in the conversation, (cid:15) u i =  Distil  BERT ( u i ) [ CLS ] , (1) by selecting the embedding of the [CLS] token, whereas to encode the character's mentions we take the mean of the embedding of the tokens inside the mention. For a mention m j = { m 0 j , m 1 j , ..., m t  1 j } of length t , (cid:15) m j = 1 t t  1 (cid:88) T =0  Distil  BERT ( u i ) m Tj . (2) We denote the collection of all mentions embeddings by M , a matrix of dimensions D (cid:15)  D l , where D l is the number of mentions to candidate characters inside the conversation. In the next section, we explain the three components of the model that take these embeddings and produce the probability of a character speaking: i) the conversation embedding module, that takes an utterance as input and produces its context-aware representation, ii) a character extraction module, that given the contextual representation of an utterance and the embedding of the candidate characters mentions, generates the logits for each candidate character entity and utterance, and iii) a sequential decoder component that learns the conversational turn patterns. Below, we define the architecture of each component in detail. The conversation embedding module consists of a Gated Recurrent Unit (Cho et al., 2014; Gers et al., 1999),  GRU , that encodes the content of the conversation, where h i is the GRU's hidden state of dimension D (cid:15) which is randomly initialised at the beginning of the sequence. This module processes the GRU's hidden states to extract the candidate character's logits. We take the dot product of the utterance embedding, processed by a fully connected network, with the mention embedding matrix, M , to obtain the logits, l i =  FCN ( h i )  M , (4) where l i , has dimensions D l . This for whole conversation results in L of dimension D n  D l and are denoted by Mention Logits in Figure 1. We can now combine the logits of different mentions that belong to the same character entity by max-pooling over character entities to get the Entity Logits, E with dimensions D n  D k . Implicitly, the model defined above assumes that labels are independent of each other, and that therefore the likelihood of a sequence of labels in a conversation, y , can be expressed as the product of utterance-wise likelihoods, However, characters speaking in a conversation follow certain turn-taking patterns that are common through literature, such as a two-party conversation in which the dialogues move back and forth between two characters. We add a linear chain Conditional Random Field (CRF) (Lafferty et al., 2001) to our model, to maximise the likelihood of a sequence of characters and relax the label indepen-dence assumption by allowing a target to depend on its immediate predecessor. A CRF models the sequential likelihood as a combination of element-wise prediction, and a pairwise interaction term that models the probability of transitioning from label y i to label y j . In our particular implementation, the element-wise predictions are the Entity Logits, E , and the pairwise interaction will be learned parameters, P ( y | u ) = exp (cid:32) N (cid:88) n =0 E n ( y n ) + N  1 (cid:88) n =0 V y n ,y n +1 (cid:33) /Z, (6) where Z represents the normalization factor, and V is generally a D k  D k dimensional matrix of learned weights known as transition matrix. In our use-case, there is no specific label ordering that can generalise to unseen novels, and we thus reduce the degrees of freedom of the D k  D k transition matrix to two: the value of the diagonal, and the value of the off-diagonal elements. The first one controls the probability of the same speaker to continue speaking, whereas the second one varies the probability of a change in speaker. Note that this implies that we do not need to constrain the number of speaking characters. At inference, we find the most likely sequence of characters using the Viterbi algorithm (Viterbi, 1967). In this section, we present both the evaluation of each separate component and the final evaluation of the entire pipeline. To compare ground truth direct speech with our extracted quotes, we define a True Positive as an exact match between our selected text and the ground truth. With this definition, the F1-score achieved by our quote identification module is 0 . 98  0 . 01 , when evaluated against our entire dataset. We find that common errors are the identification of quoted text that has a different purpose other than direct 5825 Model F1-Score Precision Recall NER 0 . speech, such as emphasising a word, naming a title, or marking written text, such as a letter, that no one is reading out loud. We evaluate separately the effects of identifying character mentions and clustering mentions into distinct entities. Since our model aims to resolve dialogue attribution and characters that are mentioned more often tend to also speak more often, we show precision, recall and F1-score weighted by the number of times a mention appears through the text. In this way, we make sure that we are identifying the main characters in the text at the cost of missing rarer ones. The evaluation is shown in Table 2. NER and our simple rule-based model show a similar performance, although overall the rule-based system works better. Finally, we evaluate character clustering on oracle mentions using the B 3 measures of precision, recall and F1-score (Amig et al., 2009). The results are reported in Table 3. The performance of coreference resolution is significantly worse than the simple naming rules we developed. By using name compatibility we achieve a high precision but a low recall, meaning that the clusters we create tend to contain elements of the same class, but are incomplete; a character might be split into several different clusters. This is because we only cluster characters from variations of their names, and therefore all other cases shown in Table 1 are considered as separate entities. As mentioned in Section 4.2.2, the coreference resolution model fails at linking two mentions to the same character that are far apart, and therefore produces a system with lower recall. We train and evaluate the speaker attribution task on oracle direct speech, mentions and character clusters. We compute precision, recall and F1-Score, all weighted averages, of the character entities attributed to each span of direct speech. The resulting evaluation is shown in Table 4. We show a comparison with a baseline model that selects the nearest mention to either left or right of the quote. To include a thorough evaluation despite the small size of our dataset, we have trained the model in a leave-one-out fashion for all books for which we annotated more than 1 , 000 paragraphs, together with the three publicly available books released by Muzny et al. (2017). In Table 4, we report average values and standard deviation over the 11 books. Moreover, we show an ablation study in Table 5, computed on only one train, validation and test split. Next to the overall F1-Score we show the performance on the model by type of signal where a sample is: i) explicit, if the character is mentioned on the same paragraph as the quote, ii) implicit, if there is no narrator context accompanying the quote. Note that not all quotes fall in either of these categories. Finally, the entire pipeline is evaluated as a clustering overlap problem through the B 3 clustering metric. A cluster is defined by the set of quotations attributed to the same character entity. If the quote has been incorrectly identified as a quote by the model, it forms part of a misidentified cluster. On the other hand, if we haven't identified one of the true quotes, we also label it as another kind of misidentification. In Figure 2, we show a full pipeline comparison of our model to the state-of-the-art model presented in Muzny et al. (2017) 2 . Our model improves over Muzny et al. (2017) by an average of 50% in F1-2 We ran their publicly available code on our dataset, the code can be found here https://nlp.stanford.edu/ ~muzny/quoteli.html 5826 F1-Score Accuracy Explicit Accuracy Implicit Distil-BERT (DB) 0 . We also show the effect of replacing the different components with Oracle data for a subsample of the dataset in Figure 3. We can see that different components play a different role by book, whereas improving the mention extraction stage can be of crucial importance for some books (14 and 15 part of text split; and 1 and 11 part of train split), character clustering has a larger effect on others. However, the dominant effect is still the Speaker attribution model. We have presented a speaker attribution pipeline for novels that does not rely on pre-compiled lists of characters and that performs consistently across different writing styles and time periods. Our main contribution has been to develop the first deep learning model for speaker attribution, based on previous Dialogue State Tracking approaches. Our deep learning model has the flexibility to learn nuanced features from data, compared to previous work that relied on rules or hand-crafted features. Training our model on a small dataset composed of 15 different novels, we find that it outperforms the model presented in Muzny et al. (2017) by an average of 50% F1-score. In the future, we hope to improve our model by: training it on a larger and more varied dataset, and training the model on speaker attribution together with the related task of coreference resolution. We have also presented an error analysis on the different components necessary to perform the end-to-end goal of attributing characters to their direct speech utterances in novels: i) direct speech identification, ii) character extraction, and iii) speaker attribution. We have shown the need to produce literature-domain specific models targeting character extraction in order to improve the accuracy of current systems.
objective method objective objective abstain result abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain objective result abstain method method method other other other other other other method other other other other other other other other other method other other objective objective abstain abstain method method abstain method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain other abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain method method abstain abstain method abstain abstain method abstain method abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain objective objective method abstain result objective abstain abstain abstain result