Humans use language to refer to entities in the external world. Motivated by this, in recent years several models that incorporate a bias towards learning entity representations have been proposed. Such entity-centric models have shown empirical success, but we still know little about why. In this paper we analyze the behavior of two recently proposed entity-centric models in a referential task, Entity Linking in Multi-party Dialogue (SemEval 2018 Task 4). We show that these models outperform the state of the art on this task, and that they do better on lower frequency entities than a counterpart model that is not entity-centric, with the same model size. We argue that making models entity-centric naturally fosters good architectural decisions. However, we also show that these models do not really build entity representations and that they make poor use of linguistic context. These negative results underscore the need for model analysis, to test whether the motivations for particular architectures are borne out in how models behave when deployed. Modeling reference to entities is arguably crucial for language understanding, as humans use language to talk about things in the world. A hypothesis in recent work on referential tasks such as co-reference resolution and entity linking (Haghighi and Klein, 2010; Clark and Manning, 2016; Henaff et al., 2017; Aina et al., 2018; Clark et al., 2018) is that encouraging models to learn and use entity representations will help them better carry out referential tasks. To illustrate, creating an entity representation with the relevant information upon reading a woman should make it easier to  denotes equal contribution. resolve a pronoun mention like she . 1 In the mentioned work, several models have been proposed that incorporate an explicit bias towards entity representations. Such entity-centric models have shown empirical success, but we still know little about what it is that they effectively learn to model. In this analysis paper, we adapt two previous entity-centric models (Henaff et al., 2017; Aina et al., 2018) for a recently proposed referential task and show that, despite their strengths, they are still very far from modeling entities. 2 The task is character identification on multiparty dialogue as posed in SemEval 2018 Task 4 (Choi and Chen, 2018). 3 Models are given dialogues from the TV show Friends and asked to link entity mentions (nominal expressions like I , she or the woman ) to the characters to which they refer in each case. Figure 1 shows an example, where the mentions Ross and you are linked to entity 335, mention I to entity 183, etc. Since the TV series revolves around a set of entities that recur over many scenes and episodes, it is a good benchmark to analyze whether entity-centric models learn and use entity representations for referential tasks. Our contributions are three-fold: First, we adapt two previous entity-centric models and show that they do better on lower frequency entities 1 Note the analogy with traditional models in formal linguistics like Discourse Representation Theory (Kamp and Reyle, 2013). 2 Source code for our model, the training procedure and the new dataset is published on https://github.com/ amore-upf/analysis-entity-centric-nns . (a significant challenge for current data-hungry models) than a counterpart model that is not entity-centric, with the same model size. Second, through analysis we provide insights into how they achieve these improvements, and argue that making models entity-centric fosters architectural decisions that result in good inductive biases. Third, we create a dataset and task to evaluate the models' ability to encode entity information such as gender, and show that models fail at it. More generally, our paper underscores the need for the analysis of model behavior, not only through ablation studies, but also through the targeted probing of model representations (Linzen et al., 2016; Conneau et al., 2018). Modeling. Various memory architectures have been proposed that are not specifically for entity-centric models, but could in principle be employed in them (Graves et al., 2014; Sukhbaatar et al., 2015; Joulin and Mikolov, 2015; Bansal et al., 2017). The two models we base our results on (Henaff et al., 2017; Aina et al., 2018) were explicitly motivated as entity-centric. We show that our adaptations yield good results and provide a closer analysis of their behavior. Tasks. The task of entity linking has been formalized as resolving entity mentions to referential entity entries in a knowledge repository, mostly Wikipedia (Bunescu and Pasca, 2006; Mihalcea and Csomai, 2007 and much subsequent work; for recent approaches see Francis-Landau et al., 2016; Chen et al., 2018). In the present entity linking task, only a list of entities is given, without associated encyclopedic entries, and information about the entities needs to be acquired from scratch through the task; note the analogy to how a human audience might get familiar with the TV show characters by watching it. Moreover, it addresses multiparty dialogue (as opposed to, typically, narrative text), where speaker information is crucial. A task closely related to entity linking is coreference resolution , i.e., predicting which portions of a text refer to the same entity (e.g., Marie Curie and the scientist ). This typically requires clustering mentions that refer to the same entity (Pradhan et al., 2011). Mention clusters essentially correspond to entities, and recent work on coreference and language modeling has started exploiting an explicit notion of entity (Haghighi and Klein, 2010; Clark and Manning, 2016; Yang et al., 2017). Previous work both on entity linking and on coreference resolution (cited above, as well as Wiseman et al., 2016) often presents more complex models that incorporate e.g. hand-engineered features. In contrast, we keep our underlying model basic since we want to systematically analyze how certain architectural decisions affect performance. For the same reason we deviate from previous work to entity linking that uses a specialized coreference resolution module (e.g., Chen et al., 2017). Analysis of Neural Network Models. Our work joins a recent strand in NLP that systematically analyzes what different neural network models learn about language (Linzen et al., 2016; Kadar et al., 2017; Conneau et al., 2018; Gulordava et al., 2018b; Nematzadeh et al., 2018, a.o.). This work, like ours, has yielded both positive and negative results: There is evidence that they learn complex linguistic phenomena of morphological and syntactic nature, like long distance agreement (Gulor-dava et al., 2018b; Giulianelli et al., 2018), but less evidence that they learn how language relates to situations; for instance, Nematzadeh et al. (2018) show that memory-augmented neural models fail on tasks that require keeping track of inconsistent states of the world. We approach character identification as a classification task, and compare a baseline LSTM (Hochreiter and Schmidhuber, 1997) with two models that enrich the LSTM with a memory module designed to learn and use entity representations. LSTMs are the workhorse for text processing, and thus a good baseline to assess the contribution of this module. The LSTM processes text of dialogue scenes one token at a time, and the output is a probability distribution over the entities (the set of entity IDs are given). The BILSTM model is depicted in Figure 2. It is a standard bidirectional LSTM (Graves et al., 2005), with the difference with most uses of LSTMs in NLP that we incorporate speaker information in addition to the linguistic content of utterances. The model is given chunks of dialogue (see Appendix for hyperparameter settings such as the chunk size). At each time step i , one-hot vectors for token t i and speaker entities s i are embedded Joey you ... softmax W t W e Joey: think Joey: you Joey: love { ... ... ... via two distinct matrices W t and W e and concatenated to form a vector x i (Eq. 1, where (cid:107) denotes concatenation; note that in case of multiple simultaneous speakers S i their embeddings are summed). The vector x i is fed through the nonlinear activation function tanh and input to a bidirectional LSTM. The hidden state  h i of a uni directional LSTM for the i th input is recursively defined as a combination of that input with the LSTM's previous hidden state  h i  1 . For a bi directional LSTM, the hidden state h i is the concatenation of the hidden states  h i and  h i of two unidirectional LSTMs which process the data in opposite directions (Eqs. 2-4). h i = LSTM ( tanh ( x i ) ,  h i  1 ) (2)  h i = LSTM ( tanh ( x i ) ,  h i +1 ) (3) h i =  h i (cid:107)  h i (4) For every entity mention t i (i.e., every token 4 that is tagged as referring to an entity), we obtain a distribution over all entities, o i  [0 , 1] 1  N , by applying a linear transformation to its hidden state h i (Eq. 5), and feeding the resulting g i to a softmax classifier (Eq. 6). Eq. 5 is where the other models will diverge. The ENTLIB model (Figure 3) is an adaptation of our previous work in Aina et al. (2018), which was the winner of the SemEval 2018 Task 4 competition. This model adds a simple memory module that is expected to represent entities because its vectors are tied to the output classes (accordingly, Aina et al., 2018, call this module entity library ). We call this memory static', since it is updated only during training, after which it remains fixed. Where BILSTM maps the hidden state h i to class scores o i with a single transformation (plus softmax), ENTLIB instead takes two steps: It first transforms h i into a query' vector q i (Eq. 7) that it will then use to query the entity library. As we will see, this mechanism helps dividing the labor between representing the context (hidden layer) and doing the prediction task (query layer). A weight matrix W e is used as the entity library, which is the same as the speaker embedding in Eq. 1: the query vector q i  R 1  k is compared to each vector in W e (cosine), and a gate vector g i is obtained by applying the ReLU function to the cosine similarity scores (Eq. 8). 5 Thus, the query extracted from the LSTM's hidden state is used as a soft pointer over the model's representation of the entities. As before, a softmax over g i then yields the distribution over entities (Eq. 6). So, in the ENTLIB 4 For multi-word mentions this is done only for the last token in the mention. 5 In Aina et al. (2018), the gate did not include the ReLU nonlinear activation function. Adding it improved results. Our implementation differs from Aina et al. (2018) in one important point that we will show to be relevant to model less frequent entities (training also differs, see Section 4): The original model did not do parameter sharing between speakers and referents, but used two distinct weight matrices. Note that the contents of the entity library in ENTLIB do not change during forward propagation of activations, but only during backpropagation of errors, i.e., during training, when the weights of W e are updated. If anything, they will encode permanent properties of entities, not properties that change within a scene or between scenes or episodes, which should be useful for reference. The next model attempts to overcome this limitation. ENTNET is an adaptation of Recurrent Entity Networks (Henaff et al., 2017, Figure 4) to the task. Instead of representing each entity by a single vector, as in ENTLIB , here each entity is represented jointly by a context-invariant or static' key and a context-dependent or dynamic' value . For the keys the entity embedding W e is used, just like the entity library of ENTLIB . But the values V i can be dynamically updated throughout a scene. As before, an entity query q i is first obtained from the BILSTM (Eq. 7). Then, ENTNET computes gate values g i by estimating the query's similarity to both keys and values, as in Eq. 9 (replacing Eq. 8 of ENTLIB ). 6 Output scores o i are computed 6 Two small changes with respect to the original model (motivated by empirical results in the hyperparameter search) as in the previous models (Eq. 6). The values V i are initialized at the start of every scene ( i = 0 ) as being identical to the keys ( V 0 = W e ). After processing the i th token, new information can be added to the values. Eq. 10 computes this new information  V i,j , for the j th entity, where Q , R and S are learned linear transformations and PReLU denotes the parameterized rectified linear unit (He et al., 2015):  V i,j = PReLU ( QW ej + RV i,j + Sq i ) (10) This information  V i,j , multiplied by the respective gate g i,j , is added to the values to be used when processing the next ( i + 1 th ) token (Eq. 11), and the result is normalized (Eq. 12): V i +1 ,j = V j + g i,j   V i,j (11) V i +1 ,j = V i +1 ,j (cid:107) V i +1 ,j (cid:107) (12) Our adaptation of the Recurrent Entity Network involves two changes. First, we use a biLSTM to process the linguistic utterance, while Henaff et al. (2017) used a simple multiplicative mask (we have natural dialogue, while their main evaluation was on bAbI, a synthetic dataset). Second, in the original model the gates were used to retrieve and output information about the query, whereas we use them directly as output scores because our task is referential. This also allows us to tie the keys to the characters of the Friends series as in the previous model, and thus have them represent entities (in the original model, the keys represented entity types, not instances). The training and test data for the task span the first two seasons of Friends , divided into scenes and episodes, which were in turn divided into utterances (and tokens) annotated with speaker identity. 7 The set of all possible entities to refer to is given, as well as the set of mentions to resolve. Only the dialogues and speaker information are available (e.g., no video or descriptive text). Indeed, are that we compute the gate using cosine similarity instead of dot product, and the obtained similarities are fed through a ReLU nonlinearity instead of sigmoid. 7 The dataset also includes automatic linguistic annotations, e.g., PoS tags, which our models do not use. one of the most interesting aspects of the SemEval data is the fact that it is dialogue (even if scripted), which allows us to explore the role of speaker information, one of the aspects of the extralinguistic context of utterance that is crucial for reference. We additionally used the publicly available 300-dimensional word vectors that were pre-trained on a Google News corpus with the word2vec Skip-gram model (Mikolov et al., 2013a) to represent the input tokens. Entity (speaker/referent) embeddings were randomly initialized. We train the models with backpropagation, using the standard negative log-likelihood loss function. For each of the three model architectures we performed a random search ( > 1500 models) over the hyperparameters using cross-validation (see Appendix for details), and report the results of the best settings after retraining without cross-validation. The findings we report are representative of the model populations. Results. We follow the evaluation defined in the SemEval task. Metrics are macro-average F 1 -score (which computes the F 1 -score for each entity separately and then averages these over all entities) and accuracy, in two conditions: All entities , with 78 classes (77 for entities that are mentioned in both training and test set of the SemEval Task, and one grouping all others), and main entities , with 7 classes (6 for the main characters and one for all the others). Macro-average F 1 -score on all entities, the most stringent, was the criterion to define the leaderboard. Table 1 gives our results in the two evaluations, comparing the models described in Section 3 to the best performing models in the SemEval 2018 Task 4 competition (Aina et al., 2018; Park et al., Figure 5: Accuracy on entities with high ( > 1000), medium (201000), and low ( < 20) frequency. 2018). Recall that our goal in this paper is not to optimize performance, but to understand model behavior; however, results show that these models are worth analyzing, as that they outperform the state of the art. All models perform on a par on main entities, but entity-centric models outperform BILSTM by a substantial margin when all characters are to be predicted (the difference between ENTLIB and ENTNET is not significant). The architectures of ENTLIB and ENTNET help with lower frequency characters, while not hurting performance on main characters. Indeed, Figure 5 shows that the accuracy of BILSTM rapidly deteriorates for less frequent entities, whereas ENTLIB and ENTNET degrade more gracefully. Deep learning approaches are data-hungry, and entity mentions follow the Zipfian distribution typical of language, with very few high frequency and many lower-frequency items, such that this is a welcome result. Moreover, these improvements do not come at the cost of model complexity in terms of number of parameters, since all models have roughly the same number of parameters ( 3 . 3  3 . 4 million). 8 Given these results and the motivations for the model architectures, it would be tempting to conclude that encouraging models to learn and use entity representations helps in this referential task. However, a closer look at the models' behavior reveals a much more nuanced picture. Figure 6 suggests that: (1) models are quite good at using speaker information, as the best performance is for first person pronouns and determiners ( I , my , etc.); (2) instead, models do not seem to be very good at handling other contextual information or entity-specific properties, as the worst 8 See Appendix for a computation of the models' parameters. performance is for third person mentions and common nouns, which require both; 9 (3) ENTLIB and ENTNET behave quite similarly, with performance boosts in (1) and smaller but consistent improvements in (2). Our analyses in the next two sections confirm this picture and relate it to the models' architectures. We examine how the entity-centric architectures improve over the BILSTM baseline on the reference task, then move to entity representations (Sec-tion 6). Shared speaker/referent representation. We found that an important advantage of the entity-centric models, in particular for handling low-frequency entities, lies in the integrated representations they enable of entities both in their role of speakers and in their role of referents. This explains the boost in first person pronoun and proper noun mentions, as follows. Recall that the integrated representation is achieved by parameter sharing, using the same weight matrix W e as speaker embedding and as entity library/keys. This enables entity-centric models to learn the linguistic rule a first person pronoun ( I, me , etc.) refers to the speaker regardless of whether they have a meaningful representation of this particular entity: It is enough that speaker representations are distinct, and they are because they have been randomly initialized. In contrast, the 9 1st person: I , me , my , myself , mine ; 2nd person: you , your , yourself , yours ; 3rd person: she , her , herself , hers , he , him , himself , his , it , itself , its . simple BILSTM baseline needs to independently learn the mapping between speaker embedding and output entities, and so it can only learn to resolve even first-person pronouns for entities for which it has enough data. For proper nouns (character names), entity-centric models learn to align the token embeddings with the entity representations (identical to the speaker embeddings). We show this by using Representation Similarity Analysis (RSA) (Kriegesko-rte et al., 2008), which measures how topologically similar two different spaces are as the Spearman correlation between the pair-wise similarities of points in each space (this is necessary because entities and tokens are in different spaces). For instance, if the two spaces are topologically similar, the relationship of entities 183 and 335 in the entity library will be analogous to the relationship between the names Joey and Ross in the token space. Table 2 shows the topological similarities between the two spaces, for the different model types. 10 This reveals that in entity-centric models the space of speaker/referent embeddings is topologically very similar to the space of token embeddings restricted to the entities' names, and more so than in the BILSTM baseline. We hypothesize that entity-centric models can do the alignment better because referent (and hence speaker) embeddings are closer to the error signal, and thus backpropagation is more effective (this again helps with lower-frequency entities). Further analysis revealed that in entity-centric models the beneficial effect of weight sharing between the speaker embedding and the entity representations (both W e ) is actually restricted to first-person pronouns. For other expressions, having 10 As an entity's name we here take the proper noun that is most frequently used to refer to the entity in the training data. Note that for the all entities condition the absolute values are lower, but the space is much larger (over 22K pairs). Also note that this is an instance of slow learning; models are not encoding the fact that a proper noun like Rachel can refer to different people. two distinct matrices yielded almost the same performance as having one (but still higher than the BILSTM, thanks to the other architectural advantage that we discuss below). In the case of first-person pronouns, the speaker embedding given as input corresponds to the target entity. This information is already accessible in the hidden state of the LSTM. Therefore, mentions cluster into entities already at the hidden layer h i , with no real difference with the query layer q i (see Figure 7). Advantage of query layer. The entity querying mechanism described above entails having an extra transformation after the hidden layer, with the query layer q . Part of the improved performance of entity-centric models, compared to the BILSTM baseline, is due not to their bias towards entity representations' per se, but due to the presence of this extra layer. Recall that the BILSTM baseline maps the LSTM's hidden state h i to output scores o i with a single transformation. Gulordava et al. (2018a) observe in the context of Language Modeling that this creates a tension between two con-flicting requirements for the LSTM: keeping track of contextual information across time steps, and encoding information useful for prediction in the current timestep. The intermediate query layer q in entity-centric models alleviates this tension. This explains the improvements in context-dependent mentions like common nouns or second and third pronouns. We show this effect in two ways. First, we compare the average mean similarity s of mention pairs T e = { ( t k , t k (cid:48) ) | t k  e  k (cid:54) = k (cid:48) } referring to the same entity e in the hidden layer (Eq. 13) and the BILSTM ENTLIBENTNET h i h i q i h i q i 0.34 0.24 0.48 0.27 0.60 Table 3: Average cosine similarity of mentions with the same referent. Table 3 shows that, in entity-centric models, this similarity is lower in the hidden layer h i than in the case of the BILSTM baseline, but in the query layer q i it is instead much higher. The hidden layer thus is representing other information than referent-specific knowledge, and the query layer can be seen as extracting referent-specific information from the hidden layer. Figure 8 visually illustrates the divi-sion of labor between the hidden and query layers. Second, we compared the models to variants where the cosine-similarity comparison is replaced by an ordinary dot-product transformation, which converts the querying mechanism into a simple further layer. These variants performed almost as well on the reference task, albeit with a slight but consistent edge for the models using cosine similarity. No dynamic updates in ENTNET . A surprising negative finding is that ENTNET is not using its dynamic potential on the referential task. We con-firmed this in two ways. First, we tracked the values V i of the entity representations and found that the pointwise difference in V i at any two adjacent time steps i tended to zero. Second, we simply switched off the update mechanism during testing and did not observe any score decrease on the reference task. ENTNET is thus only using the part of the entity memory that it shares with ENTLIB , i.e., the keys W e , which explains their similar performance. This finding is markedly different from Henaff et al. (2017), where for instance the BaBI tasks could be solved only by dynamically updating the entity representations. This may reflect our different language modules: since our LSTM module already has a form of dynamic memory, unlike the simpler sentence processing module in Henaff et al. (2017), it may be that the LSTM takes this burden off of the entity module. An alternative is that it is due to differences in the datasets. 11 For the query layer, Eq. 13 is equivalent, with cos ( q t k , q t k (cid:48) ) . dataset for information extraction as entity linking. We leave an empirical comparison of these potential explanations for future work, and focus in Section 6 on the static entity representations W e that ENTNET essentially shares with ENTLIB . The foregoing demonstrates that entity-centric architectures help in a reference task, but not that the induced representations in fact contain meaningful entity information. In this section we deploy these representations on a new dataset, showing that they do notnot even for basic information about entities such as gender. Method. We evaluate entity representations with an information extraction task including attributes and relations, using information from an indepen-dent, unstructured knowledge basethe Friends Central Wikia. 12 To be able to use the models as is, we set up the task in terms of entity linking, asking models to solve the reference of natural language descriptions that uniquely identify an entity. For instance, given This person is the brother of Monica Geller. , the task is to determine that person refers to Ross Geller , based on the information in the sentence. 13 The information in the descriptions was in turn extracted from the Wikia. We do not retrain the models for this task in any waywe simply deploy them. We linked the entities from the Friends dataset used above to the Wikia through a semi-automatic procedure that yielded 93 entities, and parsed the Wikia to extract their attributes ( gender and job ) and relations (e.g., sister , mother-in-law ; see Appendix for details). We automatically generate the natural language descriptions with a simple pattern (Figure 9) from combinations of properties that uniquely identify a given entity within the set of Friends characters. 14 We 12 http://friends.wikia.com . consider unique descriptions comprising at most 3 properties. Each property is expressed by a noun phrase, whereas the article is adapted (definite or indefinite) depending on whether that property applies to one or several entities in our data. This yields 231 unique natural language descriptions of 66 characters, created on the basis of overall 61 relation types and 56 attribute values. Results. The results of this experiment are negative: The first column of Table 4 shows that models get accuracies near 0. A possibility is that models do encode information in the entity representations, but it doesn't get used in this task because of how the utterance is encoded in the hidden layer, or that results are due to some quirk in the specific setup of the task. However, we replicated the results in a setup that does not encode whole utterances but works with single attributes and relations. While the methodological details are in the Appendix, the gender' and job' columns of Table 4 show that results are a bit better in this case but models still perform quite poorly: Even in the case of an attribute like gender, which is crucial for the resolution of third person pronouns ( he/she ), the models' results are quite close to that of a random baseline. Thus, we take it to be a robust result that entity-centric models trained on the SemEval data do not learn or use entity informationat least as recoverable from language cues. This, together with the remainder of the results in the paper, suggests that models rely crucially on speaker information, but hardly on information from the linguistic context. 15 Future work should explore alternatives such as pre-training with a language modeling task, which KNOWN . 15 Note that 44% of the mentions in the dataset are first person, for which linguistic context is irrelevant and the models only need to recover the relevant speaker embedding to succeed. However, downsampling first person mentions did not improve results on the other mention types. could improve the use of context. Recall that the motivation for entity-centric models is the hypothesis that incorporating entity representations into the model will help it better model the language we use to talk about them. We still think that this hypothesis is plausible. However, the architectures tested do not yet provide convincing support for it, at least for the data analyzed in this paper. On the positive side, we have shown that framing models from an entity-centric perspective makes it very natural to adopt architectural decisions that are good inductive biases. In particular, by exploiting the fact that both speakers and referents are entities, these models can do more with the same model size, improving results on less frequent entities and emulating rule-based behavior such as a first person expression refers to the speaker. On the negative side, we have also shown that they do not yield operational entity representations, and that they are not making good use of contextual information for the referential task. More generally, our paper underscores the need for model analysis to test whether the motivations for particular architectures are borne out in how the model actually behaves when it is deployed. We gratefully acknowledge Kristina Gulordava and Marco Baroni for the feedback, advice and support. We are also grateful to the anonymous reviewers for their valuable comments. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 715154), and from the Spanish Ramon y Cajal programme (grant RYC-2015-18907). We are grateful to the NVIDIA Corporation for the donation of GPUs used for this research. We are also very grateful to the Pytorch developers. This paper reflects the authors' view only, and the EU is not responsible for any use that may be made of the information it contains.
abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain result result abstain other other abstain abstain other other other other other other other other objective method other method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method result abstain result objective other other other other other other