To extract the relationship between two entities in a sentence, two common approaches are (1) using their shortest dependency path (SDP) and (2) using an attention model to capture a context-based representation of the sentence. Each approach suffers from its own disadvantage of either missing or redundant information. In this work, we propose a novel model that combines the advantages of these two approaches. This is based on the basic information in the SDP enhanced with information selected by several attention mechanisms with kernel filters, namely RbSP (Richer-but-Smarter SDP). To exploit the representation behind the RbSP structure effectively, we develop a combined deep neural model with a LSTM network on word sequences and a CNN on RbSP. Experimental results on the SemEval-2010 dataset demonstrate improved performance over competitive baselines. The data and source code are available at https: //github.com/catcd/RbSP . One of the most fundamental tasks in natural language processing, as well as in information extraction, is Relation Extraction (RE), i.e., determining the semantic relation between pairs of named entities or nominals in a sentence or a paragraph. Take the following sentences from the SemEval-2010 task 8 dataset (Hendrickx et al., 2009) as examples: [ churn ] e 2 and started stirring it. (ii) The agitating [ students ] e 1 also put up a [ barricade ] e 2 on the Dhaka-Mymensingh highway. Here the nominals cream' and churn' in sentence (i) are of relation  Corresponding author Entity-Destination(e1,e2) while nominals students' and barricade' in sentence (ii) are of relations Product-Producer(e2,e1) . The research history of RE has witnessed the development as well as the competition of a variety of RE methodologies. All of them are proven to be effective and have different strengths by leveraging different types of linguistic knowledge, however, also suffer from their own limitations. Some early studies stated that the shortest dependency path (SDP) in dependency tree is usually concise and contains essential information for RE (Bunescu and Mooney, 2005; Fundel et al., 2006). By 2016, this approach became dominant with many studies demonstrating that using SDP brings better experimental results than previous approaches that used the whole sentence (Xu et al., 2015a,b; Mehryary et al., 2016; Cai et al., 2016; Le et al., 2018). However, using the SDP may lead to the omission of useful information (i.e., negation, adverbs, prepositions, etc.). Recognizing this disadvantage, some studies have sought to improve SDP approaches, such as adding the information from the sub-tree attached to each node in the SDP (Liu et al., 2015) or applying a graph convolution over pruned dependency trees (Zhang et al., 2018b). Another approach to extract the relation between two entities is using whole sentence in which both are mentioned. This approach seems to be slightly weaker than using the SDP since not all words in a sentence contribute equally to classify relations and this leads to unexpected noises (Nguyen and Grishman, 2015). However, the emergence and development of attention mechanism (Bahdanau et al., 2015) has re-vitalized this approach. For RE, the attention mechanism is capable of picking out the relevant words concerning target entities/relations, and then we can find critical words which determine primary useful semantic information (Zhou et al., 2016; Verga et al., 2018). We therefore need to determine the object of attention, i.e., nominals themselves, their entity types or relation label. However, conventional attention mechanism on sequence of words cannot make use of structural information on dependency tree. Moreover, it is hard for machines to learn the attention weights from a long sequence of input text. In this work we propose an enhanced representation for relations that combines the advantages of the above approaches. Basically, we focus on condensed semantic and syntactic information on the SDP. Compensating for the limitations of the SDP may still lead to missing information so we enhance this with syntactic information from the full dependency parse tree. Our idea is based on fundamental notion that the syntactic structure of a sentence consists of binary asymmetrical relations between words ( Nivre, 2005). Since these dependency relations hold between a head word (parent, predicate) and a dependent word (children, argu-ment), we try to use all child nodes of a word in the dependency tree to augment its information. Depending on a specific set of relations, it will turn out that not all children are useful to enhance the parent node; we select relevant children by applying several attention mechanisms with kernel filters. This new representation of relation is named Richer-but-Smarter SDP (RbSP). Recently, deep neural networks (DNNs) have been effectively used to learn robust syntactic and semantic representations behind complex structures. Thus, we propose a novel DNN framework which combines Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Convolutional Neural Networks (CNN) (LeCun et al., 1989) with a multi-attention layer. We proposed a novel representation of relation based on attentive augmented SDP that overcomes the disadvantages of traditional SDP. We improved the attention mechanism with kernel filters to capture the features from context vectors. We proposed an advanced DNN architecture that utilizes the proposed Richer-but-Smarter Shortest Dependency Path (RbSP) and other types of linguistic and architectural features. RE has been widely studied in NLP community for many years. Unsupervised (Hasegawa et al., 2004; Yan et al., 2009; Quan et al., 2014), semi-supervised (Chen et al., 2006; Carlson et al., 2010; Ammar et al., 2017) and distant supervision (Verga et al., 2018; Ji et al., 2017) methods have been proven effective for the task of detecting relations from unstructured text. However, in this paper, we mainly focus on supervised approaches, which usually have higher accuracy. In earlier RE studies, researchers focused on extracting various kinds of linguistic features, including both syntactic features and semantic cues (Chan and Roth, 2010; Nguyen and Grishman, 2014). However, all the feature-based methods depend strongly on the quality of designed features from an explicit linguistic pre-processing step. Based on the idea that SDPs contain the essential information for RE, many studies exploit it with several refinements. Typical refinements include negative sampling (Xu et al., 2015a) and BRCNN (Cai et al., 2016) which model the directed shortest path. Liu et al. (2015) suggested incorporating additional network architectures to further improve the performance of SDP-based methods, which uses a recursive neural network to model the sub-tree. Some works utilized information over the whole dependency tree, such as Li et al. (2017) used dynamic extended tree conditioned LSTM for RE and Panyam et al. (2018) exploited whole dependency graph for relation extraction in biomedical text. Recently, with the introduction and development of attention mechanism, many works tend to use whole sentence or paragraph and focus on the most relevant information using attention technique. Some studies apply a single attention layer, that focus on the word itself (Shen and Huang, 2016; Zhang et al., 2018a); word position (Zhang et al., 2017) and global relation embedding (Su et al., 2018). Other works apply several attention layers, such as word, relation and pooling attention (Wang et al., 2016), multi-head attention (Verga et al., 2018) and wordand entity-based attention (Jat et al., 2017). Luo et al. (2018) used a bidirectional Long Short-Term Memory architecture with an attention layer and a tensor layer for organizing the context information and detecting the connections between two nominals. As previously mentioned, we utilize the condensed information in the SDP to learn the relation between two nominals. The simple structure of the SDP is one of its weaknesses since there exists some useful information in dependency tree that does not appear in the SDP. This information can be leveraged to represent the relation more precisely. Two examples in Figure 1 belong to different relation types, but the paths between two nominals in these examples contain only one token ( put ). However, the meaning of token  put  in two SDPs are completely different. In this situation, it is difficult for the machine to distinguish the two shortest dependency paths from these instances. We notice that the child nodes attached to the shortest dependency paths and their dependency relation from their parent can provide supplemental information for relation classification. In the previous examples, the sub-structure   prt  up  provides semantic information about token  put  in the specific sentence to make it discriminated from the stand-alone one. Based on similar observations, we propose the idea of combining subtree information with original SDP to form a more precise structure for classifying relations. In this RbSP structure each token t is represented by itself and its attached children on the dependency tree. 4 Proposed Model The overall architecture of our proposed model is shown in Figure 2. Given a sentence and its dependency tree, we build our model on the SDP between two nominals and its directed children on the tree. Here, we mainly focus on the SDP representation, which is composed of decream put churn dobj prep:in the soured We det amod W c , b c W i nd o w p r o c e ss i n g max-pooling W f , b f SDPR e p r e s e n t a t i o n W o r d e m b e dd i n g D e p e nd e n c y e m b e dd i n g A u g m e n t e d i n f o r m a t i o n Softmax Convolved features nsubj the butter det comp A tt e n t i v e a u g m e n t a t i o n A tt e n t i v e a u g m e n t a t i o n A tt e n t i v e a u g m e n t a t i o n Output Input = SDP + its nodes' children N e u r a l N e t w o r k C o n v o l u t i o n a l Figure 2: The architecture of RbSP model for relation classification. pendency embeddings, token embeddings, and token's augmented information. After SDP representation phase, each token and dependency relation is transformed into a vector. This sequence of vectors is then fed to a convolutional neural network to capture the convolved features that can be used to determine which relation two nominals are of. The goal of this phase is to represent each component on the shortest path (dependency relation and token) by a corresponding vector. We concatenate the dependency type and dependency direction to form the embedding for a dependency relation, a non-linear transformer is followed to produce the final D -dimensional representation d i  RD of i-th dependency relation as follow: d i = tanh (cid:16)h d typi  d diri i W d + b d (cid:17) (1) where d typ  R dim typ and d dir  R dim dir are dependency type and direction respectively; W d  R ( dim typ + dim dir )  D and b d  RD are trainable parameters of the network. For token representation, as mentioned above, we assume that each token should be interpreted by itself and its children. Then, the word information t i of each token on the SDP is concatenated with its attentive augmented information a i based on the attached children (which is calculated by Multi-layer attention with Kernel filters, see Section 4.2). In this work, we utilize four types of embeddings to represent the word information of each token, including:  Pre-trained fastText embeddings (Bo-janowski et al., 2017): which learned the word representation based on its external context. Character-based embeddings : we use an internal LSTM to learn the information about word morphology (like the prefix or suffix). POS tag embeddings : we embed the token's grammatical tag using a randomly initialized look-up table and update this parameter on model learning phase. WordNet embeddings : which is in form of a sparse vector that figure out which basic WordNet synsets the token belongs to. To take advantage of the original sentence sequence information, we use a recurrent neural network with LSTM units to pick up the information along the sentence S = { t i } ni =1 as follow: H =  biLSTM( S ) = { h i } ni =1 (2) Each token t i is then augmented by the corresponding hidden state h i from H . Finally, this concatenation is transformed into an X dimensional vector to form the representation x i  RX of the token. I.e., x i = tanh ([ t i  a i  h i ] W x + b x ) (3) where W x and b x are trainable parameters of the network. To capture the appropriate augmented information from the child nodes of each token, we propose a novel multi-layer attention with kernel filters architecture. As illustrated in Figure 3, we employ two sequential attention layers on the children of Token embedding Dependency embedding d 1 d 2  h 2  s 2  h 1  s 1 Kernel filters M u l t i l a y e r a tt e n t i o n F e a t u r e s e l e c t i o n I npu t Token on SDP Distance from child Childnode m a x p oo li n g S e l f a tt e n t i o n a tt e n t i o n H e u r i s t i c Token's augmented information to its father token Figure 3: The multi-layer attention architecture to extract the augmented information from the children of a token on SDP. a token to produce children context vectors. Afterward, to utilize all informative child nodes and preserve the integrity of the word information, we capture the token's augmented information using kernel filters instead of using the average of context vectors weighted by multi-layer attention. Given a token t and their child nodes, we first represent every token by a real-valued vector to provide lexical semantic features. Token t is transformed into a token embedding vector t  R dim which is the concatenation of its word embedding and part-of-speech (POS) tag embedding. To utilize all the information in the sub-structure of token's children, we form a child node not only by its token embedding as in parent node but also by the dependency relation from its direct ancestor on the sentence's parse tree. Suppose t has a set C of M children, i.e., C = { c 1 , c 2 , ..., c M } . Our model represents each child in C with a real-valued vector c i  R dim + dim dep . To additionally capture information about the child node to the target token, we incorporate the position embeddings d i to re-flect the relative distances between the i-th child's token to the target token on the original sentence. We then apply a simple self-attentive network to child nodes { c i } Mi =1 where the attention weights are calculated based on the concatenation of themselves with parent information and distance from parent, as follow:  C = (cid:8) c i  t  d i w d (cid:9) M i =1 = (cid:8)  c i (cid:9) M i =1 e = (cid:8)  c i W e + b e (cid:9) M i =1 = (cid:8) e i (cid:9) M i =1  si = exp( e i ) P Mk =1 exp( e k ) (4) where  denotes the concatenation operation; w d  R dim d is the base distance embedding; W e  R (2 dim + dim dep + dim d )  1 and b e  R are weight and bias term. The self-attentive context vector a s of the target token is the weighted sum of the self-attentive children context vectors based on the weights as follows: c si =  si c i a s = X i c s i (5) We observe that the importance of a child node to the parent node depends on the distance between them on the original sentence. Therefore, we apply a heuristic attentive layer on the self-attentive children context vectors based on the distances d 1 , d 2 , ..., d M to keep track of how close each child is to the target token. We heuristically choose the activation function for the distances d 1 , d 2 , ..., d M as f ( d ) = d 2 with  =  0 . 03 , and a softmax layer is followed to calculate the heuristic attention weight. I.e.,  hi = exp( d 2 i ) P Nk =1 exp( d 2 k ) c hi =  hi c i a h = X i c hi (6) The multi-attentive context vector a h is a synthetic representation of all child nodes with the target token node taken into account. Since the child nodes are usually distinct from each other, an average vector is not suitable to represent the children information. We propose to use the kernel filters to capture the relevant and important information from the output of the multi-attention layer. K kernel filters are applied to each child's attentive vector to produce K features from each child. I.e., F = n ReLU (cid:16) c hi W f + b f (cid:17)o M i =1 (7) where W f  R (2 dim + dim dep + dim d )  K is the weight of K kernel filters; and b f  RK is bias term. Finally, to produce the final augmented information a , we apply a max-pooling (Boureau et al., 2010) layer to the feature matrix F and select the most important features as follow: a = (cid:8) max (cid:0) F  k (cid:1)(cid:9) K k =1 (8) 4.3 CNN on RbSP After SDP representation layer, the input SDP is transformed into: SDP = h x 1 ,  d 1 , x 2 , ..., x N  1 ,  d N  1 , x N i (9) where the over arrow on d i denotes the direction of the dependency relation. We build the CNN model on this SDP ; our model is similar to the model of Xu et al. (2015a). In general, let us define the vector x i : i + j as the concatenation of j tokens and j  1 dependency relation between them. I.e., x i : i + j = x i  d i  x i +1  ...  d i + j  2  x i + j  1 (10) The convolution operation with region size r applies k filters to all possible window of r successive tokens to produce convolved feature map. We then gather the most important features by applying a max pooling (Boureau et al., 2010) layer over the entire feature map. I.e., the convolutional layer computes the i-th element of the convolved feature vector f as follows: f i = max 0  j  N  r +1 [ x j : j + r W c + b c ] i (11) where W c  R ( rX +( r  1) D )  k and b c  R k are the weight matrix and bias vector of the convolutional layer. The output f of the convolutional layer is then fed to a softmax classifier to predict a ( K + 1) -class distribution over labels  y :  y = softmax ( fW y + b y ) (12) where W y and b y are parameter of the network to be learned. The proposed model can be stated as a parameter tuple  = ( W , b ) . To compute the model parameters  , we define the training objective for a data sample as: L (  ) =  KX i =0 y i log  y i +  k  k 2 (13) where y  { 0 , 1 } ( K +1) indicating the one-hot vector represented ground truth; and  is a regularization coefficient. By minimizing L (  ) using mini-batch gradient descent (GD) with Adam optimizer (Kingma and Ba, 2014),  is updated through neural network structures. For this paper, we directly utilize the pre-trained fastText word embeddings model (Bojanowski et al., 2017) which is trained on Wikipedia data. The look-up tables for dependency embeddings, word characters, POS tags are randomly constructed using the Glorot initializer (Glorot and Bengio, 2010) and are treated as the parameters to be learned during the training phase. Since the CNN model takes the fixed size matrix as input, we pad the inputs in each batch of data dynamically to the longest input length of the batch. We further use the batch normalization (Ioffe and Szegedy, 2015) which is able to enable higher learning rates and reduces over-fitting. During the training phase, we make use of several techniques, including: clipping the gradients if their norm exceeds a given threshold (Goldberg, 2017); applying dropout (Srivastava et al., 2014) with the probability of 0.5 on embeddings layer, CNN hidden states, and penultimate layer; and using early stopping (Caruana et al., 2001) by validation loss. Further, to reduce the impact of random effects on our model, we employ the ensemble mechanism (Krogh and Sollich, 1997). For this study, we run the model for 20 times and uses the strict majority vote to obtain the final results. Our model was evaluated on SemEval-2010 Task 8 dataset ( Hendrickx et al., 2009), which contains 10 , 717 annotated relation classification examples and is separated into two subsets: 8 , 000 instances for training and 2 , 717 for testing. We randomly split 10 percents of the training data for validation. There are 9 directed relations and one undirected Other class. We conduct the training-testing process 20 times and calculate the averaged results. For evaluation, the predicted labels were compared to the golden annotated data using standard precision (P), recall (R), and F1 score metrics. Table 1 summarizes the performance of our model and comparative models. For a fair comparison with other researches, we implemented a baseline model, in which we remove all the proposed augmented information (multi-layer attention with kernel filters and LSTM on original sentence). This baseline model is similar to the model of Xu et al. (2015a) with some technical improvements and additional information sources. It yields higher F1 than competitors which are based on SDP without any data augmentation methods. This result is also comparative when is placed next to the result of basic Attention-CNN model. The results also demonstrate the effectiveness of our proposed methods that brings an improvement of 1 . 5% in F1, compared to the baseline result. Our RbSP model yields an F1-score of 86 . 3% , outperforms other comparative models, except Multi-Att-CNN model of Wang et al. (2016) with multi-level attention CNN. However, we have tried to re-implement the Multi-Att-CNN, but we failed to reproduce the positive result in the original paper. The performance of our re-implementation is about 84 . 9% of F1. This result has a high consensus with Luo et al. (2018) since they also tried to re-build this model, and their reimplemented result is not much different from us, as 85 . 5% . It is worth to note that when comparing with another augmented method of Liu et al. (2015), our multi-layer attention with kernel filters architecture brings more significant improvement. Relatively, in comparison of efficiency of augmented methods on the baseline model, the full-tree augmentation only brings 1% improvement of F1 while our attentive augmentation boosts up to 1 . 5% . Unlike the method of using the whole subtree to supplement information for the target node, our method only uses the most relevant nodes that are direct children to represent augmented infor-Model Source of information F1 depLCNN (Xu et al., 2015a) Word embeddings, SDP, CNN 81.9 + WordNet, word around nominals 83.7 + Negative sampling 85.6 BRCNN (Cai et al., 2016) Word embeddings, SDP, LSTM, CNN 85.4 + POS, NER, WordNet embeddings, inverse SDP 86.3 DepNN (Liu et al., 2015) 200-d Gigaword embeddings, SDP, CNN 81.8 + Augmented sub-tree, Recursive Neural Network 82.8 + NER 83.6 Attention-CNN (Shen and Huang, 2016) Sentence convolution, Attention-based context 84.3 + WordNet, Words around nominals 85.9 AT-BLSTM (Luo et al., 2018) Word embeddings, Sentence attention features, Tensor feature 86.3 Multi-Att-CNN (Wang et al., 2016) Multi-Level Attention CNNs, Attention pooling 88.0  85.5  Baseline Word embeddings, POS tag, WordNet 84.8 RbSP (our model) Baseline + Augmented Information 86.3 + ensemble 86.7 Table 1: The comparison of our RbSP model with other comparative models on SemEval-2010 task 8 dataset. mation. In addition, our method further focuses on the most important children through two attention layers. We also observe that during many training-testing processes, the results may vary. The standard deviation of 20 runs is about 0 . 27 . We perform the ensemble strategy by majority voting on the results of 20 runs, and it drives our model to achieve a better result of 86 . 7% . This result is outperformed other comparative models. Figure 4 shows the changes in F1 when removing each proposed component from the RbSP model. The F1 reductions illustrate the contributions of all proposals to the final result. However, the impact levels vary with different components. Between two proposed component, the multi-layer attention with kernel filters (augmented information) plays a vital role when contributing 1 . 22% to the final performance while the contribution of the LSTM on the original sentence is 0 . 33% . An interesting observation comes from the inte-rior of the multi-layer attention with kernel filters. The impact of removing the whole augmented information is much higher than the total impact of removing multi-layer attention or kernel filters ( 1 . 22 vs. 0 . 42+0 . 18 = 0 . 6 ). These results demonstrate that the combination of constituent parts is thoroughly utilized by our sequential augmented architecture. Another experiment is on investigating the meaning of each attention component. The result lightly reduces when we remove the self-attention or heuristic attention component. The results also prove that our proposed heuristic attention method is simple but effective. Its improvement is equivalent to the self-attention which is a complex attention mechanism. Among the input of multi-layer attention, the word embedding has a great influ-ence on the model performance. However, children POS tag and relation to parent are also essential components to have the good results. We studied model outputs to analyze system errors in the cases of using the baseline model and using the proposed model with RbSP representation. In Figure 5, we considered four types of errors: If the model makes a wrong decision and labels an Other relation (negative) as an actual relation (positive), it indicates 1 FP (False Positive) error. Vice versa, if it labels an actual relation as Other , it brings 1 FN (False Negative). In the case that model confused between two types of relations, the model will be penalized twice, with 1 FP and 1 FN . Direction error, i.e., the model predicts the relation correctly but its direction wrongly, also brings 1 FP and 1 FN . The proportions of the left and the right of Figure 5 are quite consistent. In which, RbSP seems to have the most impact on determining whether an instance is positive or negative. RbSP also changes the decision of the relation type in quite many cases. It also influences the decision-making about relation's directionality, but not much. Totally, the use of RbSP helps to correct more than 150 errors of the baseline model. However, it also yields some new errors (about 70 errors). Therefore, the difference of F 1 between the baseline model and our RbSP model is only 1 . 5% , as stated in table 1. Table 2 gives some realistic examples of different results when using the RbSP and not. We observed that the baseline model seems to be stuck in over-fitting problem, for examples, it classified all SDP with prep:with as Instrument-Agency and all SDP with prep:in as Member-Collection (exam-44% 19% 31% 6% RbSP Improvements Removing wrong relations Finding new relations Fixing relation type Fixing relation direction 39% 18% 40% 3% RbSP Breakdowns New wrong relations Missing relations Wrong relation type Wrong relation direction Figure 5: Comparing the effects of using RbSP in two aspects, ples 1  2 ). RbSP is really useful for solving these cases partly since it uses attentive augmentation information to distinguish the same SDP or the same preposition with different meanings. RbSP is also proven to be stronger in examples 3  4 to find new results and examples 5  7 to fix wrong results. In our statistic, the use of RbSP bring the big advantage for the relations Component-Whole , Message-Topic , Entity-Destination , Product-Producer and Instrument-Agency . The results are almost constant for Member-Collection relations. Vice versa, we regret to state that using RbSb brings some worse results (examples 8  11 ), especially for Cause-Effect and Content-Container relations. Many errors seem attributable to the parser or our model's limitations that still cannot be overcome by using the RbSP (Examples 12  13 ). We listed here some highlight problems to prioritize future researches (a) information on the SDP and its child nodes is still insufficient or redundant to make the correct prediction, (b) the direction of relations is still challenging since some errors appeared because we predict the relation correctly but its direction wrongly (c) the over-fitting problem (leading to wrong prediction FP ) and (d) lacking in generality (cannot predict new relation FN ). In this paper, we have presented RbSP, a novel representation of relation between two nominals in a sentence that overcomes the disadvantages of traditional SDP. Our RbSP is created by using multilayer attention to choose relevant information to augment a token in SDP from its child nodes. We also improved the attention mechanisms with kernel filters to capture the features on the context vector. We evaluated our model on SemEval-2010 task 8 dataset, then compared the results with very recent state-of-the-art models. Experiments were also constructed to verify the rationality and effectiveness of each of the model's components and information sources. The results demonstrated the advantage and robustness of our model, includes the LSTM on the original sentence, combination of self-attention and heuristic mechanisms and several augmentation inputs as well. The analysis of the results still points our some weaknesses of the model. We aim to address them and further extensions of our model in future works. We released our source code and data on the public repository to support the re-producibility of our work and facilitate other related studies. This research was supported by an EPSRC Experienced Researcher Fellowship (N. Collier: EP/M005089/1). We also thank the anonymous reviewers for their comments and suggestions.
abstain abstain objective abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain objective method abstain abstain objective method abstain abstain abstain objective result objective abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method method abstain method method method abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain method method method abstain method other abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain other abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain objective objective result result abstain objective result objective abstain other other