Opinion target extraction and opinion words extraction are two fundamental subtasks in Aspect Based Sentiment Analysis (ABSA). Recently, many methods have made progress on these two tasks. However, few works aim at extracting opinion targets and opinion words as pairs. In this paper, we propose a novel sequence labeling subtask for ABSA named TOWE (Target-oriented Opinion Words Ex-traction), which aims at extracting the corresponding opinion words for a given opinion target. A target-fused sequence labeling neural network model is designed to perform this task. The opinion target information is well encoded into context by an Inward-Outward LSTM. Then left and right contexts of the opinion target and the global context are combined to find the corresponding opinion words. We build four datasets for TOWE based on several popular ABSA benchmarks from laptop and restaurant reviews. The experimental results show that our proposed model outperforms the other compared methods significantly. We believe that our work may not only be helpful for downstream sentiment analysis task, but can also be used for pair-wise opinion summarization. Sentiment analysis, also known as opinion mining (Pang and Lee, 2007; Liu, 2012), has drawn increasing attention of researchers and industries in recent years. It can provide valuable information from user-generated reviews. However, sentiment analysis at sentence level or document level sometimes cannot provide more detailed information, thus a finer-grained task, Aspect-Based Sentiment Analysis (ABSA) (Pontiki et al., 2014), is proposed to identify the opinions of a specific target or aspect  Corresponding author. in reviews. ABSA consists of multiple subtasks including aspect category detection, opinion target extraction, aspect level sentiment classification etc. Opinion target extraction (OTE) and opinion words extraction (OWE) are two such fundamental subtasks. Opinion targets, sometimes called aspect terms, are the words or phrases in the sentence representing features or entities towards which users show attitude. Opinion words (or opinion terms) refer to those terms used to express attitude explicitly. For example, in the sentence  The menu is limited but almost all of the dishes are excellent. , the words  menu  and  dishes  are two opinion targets, and the words  limited  and  excellent  are opinion words. More examples can be found in the upper part of Figure 1. Recently, a great number of works based on neural networks have been done on these two subtasks (Liu et al., 2015; Poria et al., 2016; Xu et al., 2018). Furthermore, some works also integrate the two subtasks into a multi-task learning architecture to extract them jointly, which achieves great progress on both subtasks (Wang et al., 2016, 2017; Li and Lam, 2017). However, the extracted opinion targets and opinion words are not in pairs and the correspondence is not extracted. For instance, in the example sentence, (cid:104) menu:limited (cid:105) and (cid:104) dishes:excellent (cid:105) are two opinion pairs. Obviously, extracting them as pairs is significant for ABSA. Additionally, in Figure 1, the list of pairs extracted from the example review can be considered to be an extractive pair-wise opinion summarization. Considering the significance of the pairs in reviews and promising results of targets extraction in previous works, in this paper, we propose a new subtask for ABSA named TOWE (Target-oriented Opinion Words Extraction). Given a review and a target in the review, the objective of TOWE is to extract the corresponding opinion words describing or evaluating the target from the review. Then, TOWE can form pairs of the given target and its corresponding opinion words. Motivated by the success of neural networks in natural language processing, we design a powerful sequence labeling neural network model to perform TOWE. The task TOWE aims to extract the target-oriented opinion terms. In the same review, for different targets, the model needs to output different results. Therefore, a core challenge is the learning of target-specific context representations. We design a neural encoder to incorporate target information and generate the target-fused context. To be specific, we propose an Inward-Outward LSTM to pass target information to the left context and the right context of the target respectively. Then we combine the left, right and global context to encode the sentence and make sequence labeling. It is essential and reasonable to formulate TOWE as a sequence labeling task because some opinion terms include several words and one opinion target may correspond to multiple opinion terms. We try two different decoding strategies in the experiment. We propose a sequence labeling subtask for ABSA: TOWE (Target-oriented Opinion Words Extraction), which can offer assistance and interpretability for downstream tasks in ABSA. We design a novel sequence labeling neural network model to perform TOWE. It can generate target-specific context representations for different targets in the same review. We build four datasets from different domains serving as a benchmark for future works. We conduct extensive experiments on these datasets, and the results show that our model could significantly exceed a variety of baselines. A lot of works have been carried out for Opinion Targets Extraction. Traditional methods can be categorized into unsupervised/semi-supervised methods (Hu and Liu, 2004; Zhuang et al., 2006; Qiu et al., 2011) and supervised methods (Jakob and Gurevych, 2010; Shu et al., 2017). Recently, deep learning methods have also made progress in this task. Liu et al. (2015) apply a recurrent neural network with pre-trained word emebddings to solve this task. Yin et al. (2016) exploit a CRF with dependency-paths enhanced word embeddings for aspect term extraction. Poria et al. (2016) use a deep convolutional neural network (CNN) and Xu et al. (2018) propose a CNN model with double embeddings. Some works extract the targets and opinion words jointly as a co-extraction strategy. Qiu et al. (2011) propose double propagation to expand opinion targets and opinion words lists in a bootstrapping way. Liu et al.(2013) extract the targets and opinion words jointly with modeling the relation from a statistical word alignment model. This co-extraction strategy can also be adopted in neural networks with multi-task learning (Wang et al., 2016, 2017; Li and Lam, 2017). However, in all these works, the extracted targets and opinion words are separated. In the literature, only a few works discussed opinion pairs. Hu and Liu (2004) use the distance information and recognize the nearest adjective of target as the opinion words. Zhuang et al. (2006) utilize lexicons and human-built word lists to extract the targets and opinion words in the corpus, and then identify valid feature-opinion pairs with syntactic rule templates based on dependency parsing trees. However, these two methods heavily depend on the external resources such as parsers or lexicons and the performance of these approaches relies on the quality of parsing result. By contrast, our model is a purely data-driven supervised learning method and does not need any external linguistic knowledge, lexicons or handcrafted templates. Moreover, in these two methods, the process of detecting opinion words and the process of discovering correspondence is separated into two tasks, which suffers from error propagation. Our model for TOWE aims at detecting the corresponding opinion words in one step with sequence labeling. Given a sentence s = { w 1 , w 2 , . . . , w i , . . . , w n } consisting of n words, and a opinion target t in the sentence, the task is to make sequence labelling on the sentence to extract the target-oriented opinion words. We use the BIO tagging scheme (Ramshaw and Marcus, 1995) on this task. For each word w i in the sentence s , it should be tagged as y i  { B, I, O } (B: Beginning, I: Inside, O: Others). For example, for different opinion targets, the sentence  Waiters are very friendly and the pasta is out of this world .  is tagged in w i /y i style as follows: 1. Waiters/O are/O very/O [ friendly/B ] and/O the/O pasta/O is/O out/O of/O this/O world/O ./O (Given opinion target: waiter , extract friendly as corresponding opinion word). 2. Waiters/O are/O very/O friendly/O and/O the/O pasta/O is/O [ out/B of/I this/I world/I ] ./O (Given Opinion target: pasta , extract out of this world as corresponding opinion words). Figure 2 shows the framework of our methods, which follows an encoder-decoder architecture. We propose a target-fused encoder to incorporate the target information into context and learn target-specific context representations, then pass them to the decoder for sequence labeling. In the target-fused encoder, we first use an Inward-Outward LSTM to model the left context and right context of the target, then combine them with the global context. In the decoder, we can adopt two different decoding strategies. We present the details of each component in the following sections. We first generate the input vectors for each word by using a embedding lookup table L  R d | V | where d is the embedding dimension and | V | is the vocabulary size. The embedding lookup table will map s = { w 1 , w 2 , . . . , w t , . . . , w n } to a sequence of vectors { e 1 , e 2 ,    , e i , . . . , e n } as words representations where e i  R d . Typically, neural sequence labeling models use recurrent neural networks, such as LSTM (Hochre-iter and Schmidhuber, 1997) or BiLSTM, to model the sentence. However, merely using BiLSTM to model the whole sentence is totally target-independent. For the different target terms in the same sentence, BiLSTM outputs equal representation and cannot generate target-specific results. As mentioned before, the core challenge of TOWE is the learning of target-specific context representations. It is evident that different targets have different positions in the sentence and thus different contexts. So, we first split the sentence into three segments: left context { w 1 , w 2 ,    , w l } , target term { w l +1 ,    , w r  1 } and right context { w r ,    , w n } and left and right contexts are target-specific.We use a left LSTM to model the left context plus target and a right LSTM to model the target plus right context respectively. In this way the target-specific contexts could generate target-specific context representations. However, the direction of the two LSTMs is a crucial problem. We can use a simple strategy called Inward-LSTM, which follows the design of TD-LSTM (Tang et al., 2016). As Figure 2 shows, Inward-LSTM runs the two LSTMs from the two ends of the sentence to the middle target respectively. It runs the left LSTM from the first word to opinion target as a forward-LSTM and a right LSTM from the last word to the opinion target as a backward-LSTM, so we call it as Inward. This is a process of passing the context to target. We obtain left context representations HL and right context representations HR as follows: h L i =  LSTM ( h L i  1 , e i ) ,  i  [1 ,    , r  1] , (1) h R i =  LSTM ( h R i +1 , e i ) ,  i  [ l + 1 ,    , n ] . It is obvious that the words of opinion target { w l +1 ,    , w r  1 } are represented twice in the left LSTM and right LSTM. We simply average the two representations for the same word to get the representation of target words: h LR i = ( h L i + h R i ) 2 ,  i  [ l + 1 ,    , r  1] . (3) Then the context representation is: HI = { h L 1 ,    , h L l , h LR l +1 ,    , h LR r  1 , h R r ,    , h R n } . Although passing contexts to the target in Inward-LSTM is a good strategy for encoding whole sentence representation, only using this strategy is not enough for TOWE because the target information is not passed to the left and right context. For example, in the sentence I found the food to be out-standing. , the opinion target is food , the Inward-LSTM will first model outstanding and then model food . The representation of outstanding does not contain the information of food . To solve this problem, we design a novel strategy specifically for TOWE, i.e., Outward-LSTM. The idea of the Outward-LSTM is to pass the target to the context, which we believe is a better choice. As Figure 2 shows, the Outward-LSTM starts two LSTMs from the target in the middle and run towards the both ends of the sentence, which means the left LSTM is a backward LSTM and the right LSTM is a forward LSTM. We average the duplicate target hidden states and get the target-fused context representations HO = { h L 1 ,    , h L l , h LR l +1 ,    , h LR r  1 , h R r ,    , h R n } : h L i =  LSTM ( h L i +1 , e i ) ,  i  [1 ,    , r  1] , (4) h R i =  LSTM ( h R i  1 , e i ) ,  i  [ l + 1 ,    , n ] , (5) h LR i = ( h L i + h R i ) 2 ,  i  [ l + 1 ,    , r  1] . (6) This concise and reasonable strategy can solve the problems remaining in the Inward-LSTM. As we start the LSTM from the target, the target's information is fused into each word in the sentence. Also, the Outward-LSTM ensures that for different targets each word has different representations. Take the sentence Its camera is wonderful but the battery life is short ! as an example. For target camera or battery life , the target-fused representations for short are different and can generate target-specific results. We can combine the both strategy and adopt an Inward-Outward LSTM (IO-LSTM). IO-LSTM concatenates the outputs of Outward-LSTM and Inward-LSTM. The output of Outward-LSTM is crucial for incorporating target information into context, while the Inwards-LSTM is included so they can complement each other and act as a Target-specific Bidirectional LSTM. The target-fused context representations are denoted as HIO : h IO i = (cid:2) h I i ; h O i (cid:3) . To extract the target-oriented opinion words, only considering the context of each side in isolation is not enough. The left context and right context in the IO-LSTM are separated, and the left LSTM and right LSTM only share the opinion target. It is important to understand the global meaning of the whole sentence while detecting the opinion words on the left and right context. So we introduce the global context to further improve the IO-LSTM. We use a BiLSTM to model the whole sentence embeddings e = { e 1 , e 2 ,    , e i , . . . , e n } and obtain global contextualized representation HG as follows: h G i = (cid:104)  h i ;  h i (cid:105) , (8)  h i = LSTM (  h i  1 , e i ) , (9)  h i = LSTM (  h i +1 , e i ) . (10) Then we combine left-right contexts from IO-LSTM and global context, as shown in Figure 2. This enables us to obtain the final target-specific contextualized representation r for each word: r i = (cid:2) h IO i ; h G i (cid:3) . The final representation r is fused with both target information and global context information, which can be passed to the decoder for sequence labeling. Given a sequential representation r , we can use r to compute p ( y | r ) where y = { y 1 ,    , y n } are BIO-label sequence for the sentence and y i  { B , I , O } . Two different decoding policies can be adopted. The first is greedy decoding, formulated as a three-class classification problem at each position independently. We use softmax to compute the probability: p ( y i | r i ) = softmax ( W s r i + b s ) . Greedy decoding just simply selects the tag with highest point-wise probability. It does not consider the dependencies between tags but runs faster. We use the negative log likelihood (NLL) as the loss for one sentence: L ( s ) =  n (cid:88) i =1 3 (cid:88) k =1 I ( y i = k ) log p ( y i = k | w i ) . The second decoding method is to use Conditional Random Field (CRF) (Lafferty et al., 2001). CRF considers the correlations between tags in neighborhoods and score the whole sequence of tags. Specifically, we use a linear-chain CRF and score the tag sequence as conditional probability: p ( y | r ) = exp( s ( r , y )) (cid:80) y (cid:48)  Y exp( s ( r , y (cid:48) )) , (14) where Y is the set of all possible tag sequences and s ( r , y )) = (cid:80) ni ( A y i  1 ,y i + P i,y i ) is the score function. A y i  1 ,y i measures the transition score from y i  1 to y i and P i = W s r i + b s . So we use negative log likehood as the loss of the sentence: L ( s ) =  log p ( y | r ) . When given a new sentence for decoding, we will output the tag sequence that maximizes the conditional probability with Viterbi algorithm. Finally, we minimize the loss for training: J (  ) = | D | (cid:88) s L ( s ) . We build the datasets based on the SemEval challenge 2014 Task4, SemEval Challenge 2015 task 12 and SemEval Challenge 2016 task 5 (Pontiki et al., 2014, 2015, 2016). The SemEval challenge provides several datasets from restaurant and laptop do-main. These datasets are very popular benchmarks for many ABSA subtasks, including Aspect category detection, Opinion Target Extraction, Opinion Words Extraction and Target-Dependent Sentiment Analysis (TDSA). In the original datasets of SemEval challenge, the opinion targets (aspect terms) are annotated, but the opinion words and the correspondence with targets are not provided. So we annotate the corresponding opinion words for the annotated targets. Every sentence is annotated by two people, and the conflicts will be checked. Each instance of the datasets consists of a sentence, the position of the target and the positions of the corresponding opinion words. Note that we only keep the sentences that contain pairs of target and opinion words. The sentences without targets or with implicit opinion expressions are not included. Finally, we generate four datasets: 14res and 14lap from SemEval 2014, 15res from SemEval 2015 and 16res from SemEval 2016. 14res , 15res , and 16res contain reviews from restaurant domain. The sentences in 14lap come from laptop domain. The statistics of the four datasets is shown in Table 1. 4.2 Settings In our experiments, we initialize word embedding vectors with 300-dimension GloVe vectors which Dataset #sentences #targets 14res Training 1627 2643 Testing 500 865 14lap Training 1158 1634 Testing 343 482 15res Training 754 1076 Testing 325 436 16res Training 1079 1512 Testing 329 457 Table 1: Statistics of datasets. The word embeddings are fixed and not fine-tuned during the training stage. The dimension of hidden states in all the LSTM cell is set as 200. Adam (Kingma and Ba, 2015) is chosen as the optimization method with the default setting in the original paper. We randomly split 20% of the train set as dev set for tunning the hyperparameters and early stopping. Then we test the models on testing sets and the average result of five runs is reported. Precision, recall and F1 score are used as the metrics to measure the performance of models. An extracted opinion words span is regarded as a correct prediction when the starting and ending offset of the predicted span are both identical to those of a golden opinion words span. We compute Precision, Recall and F1 with the span as the unit. Since we are the first to study this sequence labeling task, there is no available sequence labeling model in the literature to be compared. Although there are a number of complicated models in TDSA, the task is different. Those TDSA models focus on sentence-level representations for sentiment classification, while for TOWE the representation on token-level representations is more crucial. Simply transferring the TDSA models for TOWE is not suitable. Except for two rule-based methods, we can only design and implement the baselines for TOWE ourselves. Our final model is the IOG encoder with a greedy decoding strategy. We compare it with the following baselines:  Distance-rule : Hu and Liu(2004) use the distance and POS tags to determine the opinion words. Following this idea, we first use the nltk toolkit to make part-of-speech tagging on each word and select the nearest adjective from the target as the corresponding opinion word. Dependecy-rule : We adopt the strategies proposed in (Zhuang et al., 2006) which uses dependency-tree based templates to identify opinion pairs. The POS tag of opinion targets and opinion words and the dependency path between them in the training set are recorded as rule templates. 1 The high-frequency dependency templates are used for detecting the related opinion words in the testing set. LSTM/BiLSTM : This method is an LSTM/BiLSTM network built on top of word embeddings proposed by (Liu et al., 2015). We pass the whole sentence into the LSTM/BiLSTM and each hidden state is fed to a softmax layer for three-class classification, which works as sentence-level opinion words extraction. Pipeline : This method combines BiLSTM and Distance-rule method in a pipelined way. We first train a sentence-level opinion words extraction model with BiLSTM and extract all the opinion words in the test sentences; then we select the closest extracted opinion words of the target as the result. Target-Concatenated BiLSTM (TC-BiLSTM) : This method incorporates the target information into sentence by concatenation. A target vector is obtained by the average pooling of target word embeddings. The word representation at each position is the concatenation of word embedding and target vector, which is then fed into a BiLSTM for sequence labeling. The main results can be found in Table 2. Note that all the neural models in Table 2 adopt greedy decoding. The performance of Distance-rule method is not satisfactory and the worst among all the methods; its recall score is especially low. IOG obtains an F1 score with a greater-than 30% improvement over the Distance-rule method. Dependency-rule method obtains a general improvement than Distance-rule, but it was still lower than the below sequence-labeling based methods. This reveals the 1 We use the parsers in spaCy: https://spacy.io Models 14res 14lap 15res 16res P R F1 P R F1 P R F1 P R F1 Distance-rule 58.39 43.59 49.92 50.13 33.86 40.42 54.12 39.96 45.97 61.90 44.57 51.83 Dependency-rule 64.57 52.72 58.04 45.09 31.57 37.14 65.49 48.88 55.98 76.03 56.19 64.62 Pipeline 77.72 62.33 69.18 72.58 56.97 63.83 74.75 60.65 66.97 81.46 67.81 74.01 LSTM 52.64 65.47 58.34 55.71 57.53 56.52 57.27 60.69 58.93 62.46 68.72 65.33 BiLSTM 58.34 61.73 59.95 64.52 61.45 62.71 60.46 63.65 62.00 68.68 70.51 69.57 TC-BiLSTM 67.65 67.67 67.61 62.45 60.14 61.21 66.06 60.16 62.94 73.46 72.88 73.10 IOG 82.85 77.38 80.02 73.24 69.63 71.35 76.06 70.71 73.25 85.25 78.51 81.69 Table 2: Main Results in terms of Precsion, Recall and F1-score. lack of robustness in rule-based approaches. The error propagation from syntactic parsers is also a reason for poor performance. The Pipeline model performs much better than rule-based methods, obtaining an especially high precision, showing that machine-learning methods can obtain better opinion words extraction. However, pipeline model is still not ideal, and the F1-score is approximately 10% lower than our proposed model in several datasets. This reflects that the distance information is not sufficient for detecting target-oriented opinion words while IOG could better handle long distance dependency problem. Also, this strategy cannot solve the cases where one target corresponds to more than one opinion term. It also suffers from error propagation. LSTM and BiLSTM are both target-independent leading to low precision, and their performance is even worse than pipeline method. IOG outperforms BiLSTM by about 15% averagely, which indicates the target information should be included. TC-BiLSTM includes the target information by concatenation and obtains better general performance than LSTM and BiLSTM. However, TC-BiLSTM is still over 10% lower than IOG and is slightly inferior to Pipeline, showing that the concatenation is not a good way to incorporate the target information for TOWE. We believe that the problem is that the concatenated target may interfere with the other targets in the same sentence. datasets from different domains compared to both the rule-based methods and neural models. We can conclude that IOG can learn target-specific representations more effectively and can better capture the correspondence between targets and opinion words. To compare the different design of our model and provide more compared models, we also report the results of the variants of our models in Table 3.  Inward-LSTM : HI computed from (1), (2), (3) are fed to the greedy decoder for sequence labeling. Outward-LSTM : HO computed from (4), (5), (6) are fed for greedy decoding  IO-LSTM : Combining Inwards-LSTM and Outwards-LSTM, HIO is obtained by concatenation of HI and HO in (7), which is then used for greedy decoding. IOG+CRF : Passing the representations r in IOG to a CRF decoder. The performance of Inward-LSTM is inferior, similar to the target-independent BiLSTM. This demonstrates that only passing the context to target is similar to not considering the target information owing to the problems we discussed before. The F1-score of Outward-LSTM exceeds that of the Inward-LSTM by more than 10%. This shows Sentence Distance-rule Dependency rule Pipeline BiLSTM TC-BiLSTM IOG The bread is top notch as well . that passing target into context is a better choice and learning the target-specific word representations is crucial. In fact, Outward-LSTM has already outperformed all the previous baselines, which indicates that this is a really good design for TOWE. IO-LSTM which combine the Inward and Outward is slightly better than Outward-LSTM, showing that Inward-LSTM can still provide supplementary information for Outward-LSTM. Through combining global context with IO-LSTM as IOG model, we roughly obtain a further 1% improvement. We also test our model with a linear Conditional-Random-Field as the decoder. CRF considers the label dependencies. It can be observed that IOG with CRF obtains a slight improvement. To demonstrate the effectiveness of our model, we pick some examples in the test dataset in 14res and show the extracted results of different models. In the first sentence, since the Distance-rule cannot extract phrases, the extraction it makes is incorrect. In addition, merely selecting the nearest adjective using the Distance-rule approach does not enable coverage in all cases, as shown in the second and third sentence (e.g., the asian and lyche ). Dependency-rule in some cases fails to extract any word owing to the error of parser and no template to match. Pipeline method has the problem that it cannot handle the cases that one target corresponds to multiple opinion terms (e.g., not great is not extracted in the fourth sentence). The drawback of BiLSTM is that it does not include target information, so it extracts both love and good in the third sentence while only love is the corresponding opinion word for drinks . Although TC-BiLSTM is a target-specific model, it tends to extract irrelevant opinion words because of the interference from concatenation. In the last two rows of Table 4, we show the same sentence with two different targets and only IOG does not make mistakes for both targets. In this paper, we propose a novel subtask for aspect-based sentiment analysis: Target-oriented Opinion Words Extraction (TOWE) which aims at extracting the corresponding opinion words for a given opinion target. We design a novel neural model IOG to solve this task. IOG can effectively encode target information into left and right context respectively. Then we combine the left and right context of the opinion target and global context for extracting the corresponding opinion word in the decoder. We contribute four datasets based on several benchmarks. The experimental results demonstrate that our model achieves the best performance across all the datasets from different domains. In future works, TOWE could be utilized to further improve the performance on downstream sentiment analysis tasks with building a more interpretable model, such as enhanced-feature or multitask learning. In addition, an end-to-end opinion extractive summary method without given golden targets is also a future work. The authors would like to thank Fang Qian and Fei Zhao for their contribution to building the datasets, and Robert Ridley for his comments on this paper. We also express the gratitude to the anonymous reviewers for their valuable feedback. This work is supported by the National Natural Science Foundation of China (No. 61672277, U1836221), the Jiangsu Provincial Research Foundation for Basic Research (No. BK20170074).
abstain abstain abstain objective abstain abstain abstain method objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain abstain abstain method objective method abstain abstain objective objective objective method result other other other other other other other other other other other other other other other other objective other objective abstain method abstain abstain abstain abstain abstain abstain method method method method method method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain method abstain abstain abstain abstain method method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain abstain method abstain abstain method abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain method method method method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective objective abstain method objective objective abstain abstain other other other