Relational triple extraction is critical to understanding massive text corpora and constructing large-scale knowledge graph, which has attracted increasing research interest. However, existing studies still face some challenging issues, including information loss, error propagation and ignoring the interaction between entity and relation. To intuitively explore the above issues and address them, in this paper, we provide a revealing insight into relational triple extraction from a stereoscopic perspective, which rationalizes the occurrence of these issues and exposes the shortcomings of existing methods. Further, a novel model is proposed for relational triple extraction, which maps relational triples to a three-dimension (3-D) space and leverages three decoders to extract them, aimed at simultaneously handling the above issues. Extensive experiments are conducted on five public datasets, demonstrating that the proposed model outperforms the recent advanced baselines. Relational triple is a common structural representation of semantic facts. A triple is always in form of (subject, relation, object), where subject and object are two entities connected by a type of predefined semantic relation. Relational triple extraction from unstructured texts is critical to understanding massive text corpora and constructing large-scale knowledge graph (Ren et al., 2017; Wei et al., 2020), which is widely concerned in recent years. Early researches (Zhou et al., 2005; Chan and Roth, 2011; Zhang et al., 2017) first recognize entities and predict the relations for each entity pair. Such approaches suffer from error propagation problem and thus recent researches (Zheng et al., 2017; Zeng et al., 2018; Fu et al., 2019;  Corresponding author: Liping Jing. Nayak and Ng, 2020; Wei et al., 2020; Liu et al., 2020) try to build a jointly-decoding schema for entities and relations. However, relational triple extraction still faces the following challenging issues:  Information loss ( I-IL ). Information loss includes entity incompleteness (Zeng et al., 2020) and entity overlapping (Zeng et al., 2018; Wei et al., 2020). Entity incompleteness (I-IL-EI) refers to that only head or tail token rather than completed entity is recognized, while entity overlapping (I-IL-EO) is that one entity belonging to multiple triples cannot be marked. Error propagation ( I-EP ). Error propagation comes from the prediction process with strict order. For examples, pipeline models (Zhang et al., 2017; Takanobu et al., 2019) recognize entities first and predict relations based on each specific entity pair. Generative models (Zeng et al., 2018, 2019) extract subject, object and relation with a predetermined order. Ignoring the interaction between entity and relation ( I-II ). Subjects (or objects) in different predefined relations should have different recognition patterns, which are not modelled when ignoring the interaction between entity and relation. To intuitively explore the above issues and address them, from a stereoscopic perspective, we map the relational triples of a text to a three-dimensional (3-D) space, which is like a cube as Figure 1. The relational triples are actually some small cubes in the whole cube. Existing researches are actually to model the cube from different perspectives and further extract the triples. Based on the representation of triples in 3-D space, three operations (i.e. slice, projection and shrinkage) Figure 1: The representation of triples in 3-D space, where a text corresponds to a cube with size being ( |L|  |T | )  ( |L|  |T | )  |R| , while each triple is mapped to a small cube with size being ( m  |T | )  ( n  |T | )  1 . are defined as Figure 2, to understand why existing methods suffer from the above issues. Furthermore, we propose a novel model for relational triple extraction, which can simultaneously handle the above issues, named StereoRel . More precisely, the cube is modelled from three perspectives, including ( x, z ) -plane projection, ( y, z ) -plane projection and z -slices, which indicates the subjects, objects and their correspondences for each predefined relation. Correspondingly, the proposed method leverages three decoders to extract relational triples in a unified model. This work has the following main contributions:  We provide a revealing insight into relational triple extraction from a stereoscopic perspective, where the occurrence of several challenging issues and shortcomings of existing methods are rationalized. We propose a novel StereoRel model for relational triple extraction, which can simultaneously reduce information loss, avoid error propagation and not ignore the interaction between entity and relation. Extensive experiments are conducted on five public datasets, demonstrating that the proposed model outperforms the recent advanced baselines. In form of (subject, relation, object), triples can naturally be mapped to a three-dimensional (3-D) space, which is elaborated in this section. Meanwhile, we define three operations (i.e. slice, projection and shrinkage) in 3-D space, to make it easy to understand the strengths and shortcomings of previous researches. Given a text L with length being |L| and a predefined relation set R having |R| relations, L may have several triples, that is, p ([ s, r, o ] |L ) where [  ] represents a collection. Each triple consists of a subject ( s ), an object ( o ) and one relation ( r ) belonging to R . Subject is one entity, that is, n-gram in L and so does object. To model p ([ s ] |L ) or p ([ o ] |L ) , the common strategy is to leverage sequence tagging on L , which has some existing strategies, such as BMES tagging (Zhang and Yang, 2018; Li et al., 2020) and start-and-end binary tagging (Wei et al., 2020; Sui et al., 2020). Anyway, there is a tag set T , and thus p ([ s ] |L ) or p ([ o ] |L ) can be represented by a vector with length being |L|  |T | . Meanwhile, due to r  R , modeling p ([ r ] |L ) can be taken as a classification task, which requires a vector with length being |R| to represent. Therefore, when modeling p ([ s, r, o ] |L ) by consid-(a) Slice . ering all possible connections, it should be equivalent to a cube with size being ( |L|  |T | ) 2  |R| in a 3-D space. As shown in Figure 1, the line segments of the cube mapping on x -axis, y -axis and z -axis are respectively regarded as the representations of subjects, objects and relations, that is, p ([ s ] |L ) , p ([ o ] |L ) and p ([ r ] |L ) . Similarly, the rectangles of the cube mapping on ( x, y ) -plane, ( x, z ) -plane and ( y, z ) -plane are respectively regarded as p ([ s, o ] |L ) , p ([ s, r ] |L ) and p ([ o, r ] |L ) . Further, each triple is mapped to a small cube in the space. Based on the stereoscopic representation of relational triples, we define the following operations. Slice , denoted as sli (  ) . As shown in Figure 2(a), when some elements (i.e. subject, object or relation) are specified, the representation space will be reduced. The operation is like slicing the cube. For instances, a specific relation corresponds to a z -slice with size being ( |L|  |T | ) 2  1 . Both subject and object being specified leads to an xy slice with size being ( m  |T | )  ( n  |T | )  |R| , which can be seen as the intersection of an x -slice and a y -slice. If subject, object or relation are Table 1: The correspondence between the issues of relational triple extraction and the operations in 3-D space. Projection , denoted as pro (  ) . As depicted in Figure 2(b), two types of projection are defined, cube-to-plane and plane-to-axis . The former seems to look at the whole cube from a certain plane. For example, in the projection from cube to ( x, y ) -plane, two triples with the same subject and object are indistinguishable. Similarly, in the projection from cube to ( x, z ) -plane, there is only subject and relation information but no object information. The later seems to look at a plane from a certain axis, such as ( x, z ) -plane to x -axis projection, where the subjects in different z -slices may have the same representation on x -axis. Hereafter, for easy reading, ( x, z ) -plane to x -axis and ( y, z ) -plane to y -axis projections are denoted as pro x (  ) and pro y (  ) respectively. The projection from cube to ( x, y ) plane is denoted as pro xy (  ) . The rest ones are similar. Shrinkage , denoted as shr (  ) . In the cube representation, each token pair is represented by an xy -slice with size being |T |  |T |  |R| . Such an xy -slice can reflect all possible entity-tagging combinations of a token pair. As described in Figure 2(c), a shrinkage over a cube only represent whether the token pair satisfies one specific entity-tagging combination, such as (start, start). Thus, the size of a shrinkage is |L|  |L|  |R| . As aforementioned, relational triple extraction faces three challenging issues: information loss ( I-IL-EI or I-IL-EO ), error propagation ( I-EP ) and ignoring the interaction between entity and relation ( I-II ). It is clear to match them to the three operations in 3-D space as Table 1. sli (  ) corresponds to the prediction process with strict order, and thus leads to error propagation. Without considering the nested entities in a text, pro xz/yz/z (  ) do not result in any problems, while pro x/y/xy (  ) is the Table 2: The analysis of previous researches from the stereoscopic perspective. opposite. Both pro x/y (  ) and pro xy (  ) lead to ignoring the interaction between entity and relation. Meanwhile, pro xy (  ) makes the triples with overlapped entities indistinguishable. The cube can be disassembled into |T |  |T | shr (  ) . Modeling only one shr (  ) will cause entity incompleteness. To get deep insights on relational triple extraction, based on the correspondence between the operations and issues, we analyze previous researches as shown in Table 2. Early researches (Zelenko et al., 2002; Zhou et al., 2005; Chan and Roth, 2011) adopt pipeline approaches, where the entities are recognized first and the relations for each entity pair are predicted. Arguing that such approaches neglect the inherent relevance between entity recognition and relation extraction, some solutions (Miwa and Bansal, 2016; Zhang et al., 2017; Takanobu et al., 2019) still extract entities and relations sequentially, but make two tasks share the same encoder. These methods model p ([ s, r, o ] |L ) as p ([ s ]  [ o ] |L ) and p ([ s i , r, o j ] |L , sli xy ( s i , o j )) , where p ([ s ]  [ o ] |L ) = p ( pro x ( pro xz ([ s, r, o ]))  pro y ( pro yz ([ s, r, o ])) |L ) . Therefore, pipeline paradigm suffers from I-EP and I-II issues. MHS (Bekoulis et al., 2018) is another two-stage method. The model recognizes entities firstly and extracts relational triples with a multi-head selection strategy on each subject, where pro x/y (  ) and sli x (  ) lead to I-EP and I-II issues respectively. In the following researches on relational triple extraction, several methods with joint decoding schema are proposed. Specifically, NovelTagging (Zheng et al., 2017) and PA-Tagging (Dai et al., 2019) achieve joint decoding by designing a unified tagging scheme and convert relational triple extraction to an end-to-end sequence tagging problem. Such a tagging schema has to model p ( pro xy ([ s, r, o ]) |L ) and thus suffers from I-IL-EO and I-II issues. CopyRE (Zeng et al., 2018) and CopyRRL (Zeng et al., 2019) leverage sequence-to-sequence model with copy mechanism. GraphRel (Fu et al., 2019) introduces graph convolutional network jointly learn entities and relations. Despite their initial success, the three methods only model p ( shr ([ s, r, o ]) |L ) and thus suffer from I-IL-EI issue. Sequence generation models, CopyRE and CopyRRL, predict triples one by one and model p ( sli xyz ( s i , r i , o i )) via p ( sli z ( r i ))  p ( sli xz ( s i , r i ) | sli z ( r i ))  p ( sli xyz ( s i , r i , o i ) | sli xz ( s i , r i )) , which leads to I-EP issue. GraphRel cannot avoid I-IL-EO and I-II issues due to its utilizing pro xy (  ) . Recently, to address I-IL issue, CopyMTL (Zeng et al., 2020) proposes a multi-task learning framework based on CopyRE, to simultaneously predict completed entities and capture relational triples. However, the model still does not solve I-EP issue. Meanwhile, entity recognition is implemented by modeling p ([ s ]  [ o ] |L ) via a standalone module, which leads to I-II issue. Following sequence-to-sequence schema, WDec and PNDec (Nayak and Ng, 2020) design specific decoder block which can generate triples with completed entities. Such models ease I-II issue, but still suffers from I-EP issue since that it models p ( sli xyz ( s i , r i , o i )) via p ( sli x ( s i ))  p ( sli xy ( s i , o i ) | sli x ( s i ))  p ( sli xyz ( s i , r i , o i ) | sli xy ( s i , o i )) . CasRel (Wei et al., 2020) regards relations as functions that map subjects to objects in a text. It is necessary to recognize subjects first and then objects, which leads to I-EP issue. To recognize subjects, p ([ s ] |L ) is modelled via p ( pro x ( pro xz ([ s, r, o ])) |L ) , where pro x (  ) leads to I-II issue. Att-as-Rel (Liu et al., 2020) models the triples by multi-head attention, where completed entities are recognized by modeling p ([ s ]  [ o ] |L ) separately and thus there is I-II issue. Similarly, TPLinker (Wang et al., 2020b) regards joint extraction as a token pair linking problem, where entity recognition is also modelled separately via p ([ s ]  [ o ] |L ) . To handle the above three issues simultaneously, we avoid to make the operations in Table 1 and try to model p ([ s, r, o ] |L ) via (cid:80) |R| i [ p ([ s, r i ] |L ) + p ([ r i , o ] |L ) + p ( shr ([ s, r i , o ]) |L )] . As depicted in Figure 3, the proposed StereoRel model first leverages BERT encoder to extract the text representation for the original text. Then, for each predefined relation, the text representation is transformed to its subject and object spaces. Based on them, three decoders are built to separately model p ( pro xz ([ s, r, o ]) |L ) , p ( pro yz ([ s, r, o ]) |L ) and p ( shr ([ s, r, o ]) |L ) . To sufficiently capture the textual information, the encoder is built by a pre-trained language model, BERT (Devlin et al., 2019). BERT encoder tok-enizes a text L using a predefined vocabulary and generates a corresponding sequence  L by concatenating a [CLS] token, the tokenized text and a [SEP] token. The detailed steps can be referred to (Devlin et al., 2019). BERT encoder will embed a text L into a matrix T  R ( |L| +2)  d b , where d b is the hidden size of BERT, and T j can be seen as the word embedding of j -th token, j  [0 , |L| + 1] . After this, for each relation r i , T is transformed to a new text representation T i  R ( |L| +2)  d r by: T i =  ( T W i + b i ) , (1) where { W i } |R| i =1  R d b  d r , { b i } |R| i =1  R 1  d r are trainable parameters and  (  ) is predetermined activation function. Subject decoder is to model (cid:80) |R| i p ([ s, r i ] |L ) , that is, ( x, z ) -plane projection p ( pro xz ([ s, r, o ]) |L ) , which recognizes the subjects for each predefined relation. For one specific relation r i , we transform its text representation to T subi  R ( |L| +2)  d e in r i 's subject space with d e being the hidden size. The transformation is implemented by T subi = T sub  q i + T sub  k i + T sub  b i , (2) T sub  q/k/b i =  ( T i W sub  q/k/b i + b sub  q/k/b i ) , (3) where T sub  q i , T sub  k i , T sub  b i are linear transformations on top of T i . T sub  q i , T sub  k i will be used by shrinkage decoder, while T sub  b i only works for subject decoder. { W sub  q i } |R| i =1 , { W sub  k i } |R| i =1 , { W sub  b i } |R| i =1  R d r  d e , { b sub  q i } |R| i =1 , { b sub  k i } |R| i =1 , { b sub  b i } |R| i =1  R 1  d e are trainable parameters and  (  ) is predetermined activation function. Based on T sub i , all possible subjects in relation r i 's subject space are recognized by a sequential conditional random field (CRF) (Lafferty et al., 2001) layer with Figure 3: The proposed StereoRel model. [ Begin , Inside , Outside ] tagging schema, where the probability of the final label sequence, y subi = [ y subi 1 , y subi 2 , ..., y subi |L| ] , is modeled as follows: P( y subi |L ) = (cid:81) |L| j =1  j ( y subi ( j  1) , y subij |L ) (cid:80) y (cid:48)  Y (cid:81) |L| j =1  j ( y (cid:48) j  1 , y (cid:48) j |L ) , (4)  j ( y,  y |L ) = exp ( T sub w crf sub y,  y + b crf sub y,  y ) , (5) where Y denotes all possible label sequence of L . w crf sub y,  y and b crf sub y,  y are trainable parameters corresponding to the label pair ( y,  y ) . Object decoder is to model (cid:80) |R| i p ([ r i , o ] |L ) , that is, ( y, z ) -plane projection p ( pro yz ([ s, r, o ]) |L ) , which recognizes the objects for each predefined relation. Similar to subject decoder, the text representation in object space, T obji  R |R| ( |L| +2)  d e , is obtained in object decoder as: T obji = T obj  q i + T obj  k i + T obj  b i , (6) T obj  q/k/b i =  ( T i W obj  q/k/b i + b obj  q/k/b i ) , (7) where T obj  q i , T obj  k i , T obj  b i are linear transformations on top of T i . { W obj  q i } |R| i =1 , { W obj  k i } |R| i =1 , { W obj  b i } |R| i =1  R d r  d e , { b obj  q i } |R| i =1 , { b obj  k i } |R| i =1 , { b obj  b i } |R| i =1  R 1  d e are trainable parameters. In like wise, objects of the i -th predefined relation are tagged as y obji = [ y obji 1 , y obji 2 , ..., y obji |L| ] via another CRF layer as: P( y obji |L ) = (cid:81) |L| j =1  j ( y obji ( j  1) , y objij |L ) (cid:80) y (cid:48)  Y (cid:81) |L| j =1  j ( y (cid:48) j  1 , y (cid:48) j |L ) , (8)  j ( y,  y |L ) = exp ( T obj w crf obj y,  y + b crf obj y,  y ) , (9) where w crf obj y,  y and b crf obj y,  y are trainable parameters. To extract the correspondences between subjects and objects, shrinkage decoder is leveraged to model (cid:80) |R| i p ( shr ([ s, r i , o ]) |L ) , where each element of shr (  ) denotes whether the corresponding token pair is one specific position of a (subject, object) pair, such as (start, start) or (end, end). To model this, a pair-wise classification function f is established as: p shrijj (cid:48) = f ( T subij , T obj ij (cid:48) ) , (10) indicating the probability that j -token and j (cid:48) -token is the specifc position of a (subject, object) pair, which satisfies the i -th predefined relation. We design the function as follows: p shrijj (cid:48) =  ( p sub  obj ijj (cid:48) , p obj  sub ijj (cid:48) ) , (11) p sub  obj ijj (cid:48) = softmax j (  ( T sub  q ij , T obj  k ij (cid:48) )) , (12) p obj  sub ijj (cid:48) = softmax j (cid:48) (  ( T obj  q ij (cid:48) , T sub  k ij )) , (13) where  (  ) is implemented by dot product or neural network to provide an initial probability. p sub  obj ijj (cid:48) and p obj  sub ijj (cid:48) respectively indicate the probability distributions for a subject searching for its objects and an object searching for its subjects, which are integrated via a predetermined function  (  ) , such as minimum, maximum and multiplication. Subject and object decoders are learned by text-level log-likelihood loss, while shrinkage decoder is learned by token-level binary cross-entropy loss. Thus, the unified model is learned by a combined loss function L total = L sub + L obj + L shr , where L sub =  |R| (cid:88) i log (P( y subi |L )) , (14) L obj =  |R| (cid:88) i log (P( y obji |L )) , (15) L shr =  |R| (cid:88) i |L| (cid:88) j |L| (cid:88) j (cid:48) (cid:104)  p shrijj (cid:48) log ( p shrijj (cid:48) )+ (1   p shrijj (cid:48) ) log (1  p shrijj (cid:48) ) (cid:105) . (16) The relational triples can be inferred based on the three decoders. Concretely, for each predefined relation r i , the subjects and objects can be obtained by y subi and y obji respectively. For the subject s ij and object o ij (cid:48) with ( j -th token, j (cid:48) -th token) satisfying the specific position, if p shrijj (cid:48) is greater than a predetermined threshold  , ( s ij , r i , o ij (cid:48) ) will be extracted as a relational triple. To evaluate the proposed StereoRel model, we conduct a performance comparison on five public datasets in this section. Evaluation Metrics and Datasets . Generally, the performance on relational triple extraction is evaluated by precision (Pre.), recall (Rec.) and F1-score (F1) , where a triple is regarded as correct if subject, relation and object are all matched. Notably, in previous works, there are two evaluation modes: Partial Match and Exact Match . The former holds that subject (or object) is correct as long as its head or tail is correct, while the latter requires it to be recognized completely. To properly compare our model with various baselines, benchmark datasets are selected for the two modes separately. Concretely, we utilize NYT (Riedel et al., 2010), WebNLG (Gardent et al., 2017), NYT10 (Takanobu et al., 2019) and NYT11 (Takanobu et al., 2019) datasets for Partial Match , while NYT (Riedel et al., 2010) and Wiki-KBP (Dai et al., 2019) datasets for Exact Match . The details are shown in Table 3. The splits of validation set are the same as previous researches. Implementation Details . For making a fair comparison, we utilize the cased BERT-base 1 model in our experiments, which is the same as CasRel (Wei et al., 2020) and TPLinker (Wang et al., 2020b), and thus d b = 768 . Adam optimizer (Kingma and Ba, 2015) is utilized to train the proposed method with initial learning rate being 1e-5. The hidden size d r , d e are set as 64, 32. The threshold  is tuned for each relation and determined by the validation set. (  ) is set as relu activation function. (  ) is set as dot product. (  ) is set as the multiplication function. 1 https://storage.googleapis.com/bert_ models/2018_10_18/cased_L-12_H-768_A-12.zip Table 4: Performance comparison by Partial Match on NYT Model Pre. We employ some recent advanced methods as baselines, mainly including the models analyzed in Table 2. Table 4, 5, 6 and 7 report the results of our method against the baselines for Partial Match evaluation mode, and Table 8 and 9 report the results for Exact Match . The models before CasRel do not employ BERT encoder and the rest does. As aforementioned in Table 2, existing models do not handle three challenging issues simultaneously, while our proposed StereoRel model does. Among the baselines, Att-as-Rel is the first work to extract triples for each predefined relation with no I-IL and I-EP issues, and thus achieves a huge performance improvement compared with previous methods. Based on BERT encoder, the performance on relational triple extraction has been further improved by CasRel and TPLinker. Due Table 6: Performance comparison by Partial Match on NYT10 Model Pre. to no I-EP issue, TPLinker outperforms CasRel. However, TPLinker still suffers from I-II issue. Our proposed StereoRel model further considers it and achieves a better performance. From the results, comparing with the second best baseline, the performance improvement of the existing best baseline on the five datasets were 2.5%, 0.1%, 1.4%, 0.1% and 1.5% respectively, in terms of F1-score . Our model obtains performance gain about 0.3%, 0.2%, 0.2%, 0.7% and 0.6% in terms of the best baseline. It can be seen that the improvement is satisfied. For relational triple extraction, from the stereoscopic perspective, there are the following two aspects worthy of discussion. The first one is about learning strategy. Most of previous studies and ours employ binary cross-entropy loss to learn the models. However, since the label space of relational triple in 3-D space is huge, binary cross-entropy is available but not necessarily optimal. Meanwhile, cross-entropy is permutation-sensitive loss function (Sui et al., 2020), which is incompatible with generative models (Zeng et al., 2018, 2019, 2020) since it is necessary to predetermine extraction order of multiple triples. To this question, CGT (Ye et al., 2020) incorporates contrastive learning strategy and SPN (Sui et al., 2020) transforms relational Table 8: Performance comparison by Exact Match on NYT Model Pre. triple extraction into set prediction problem learned by bipartite matching loss. These ideas may be introduced in the future. The second one is to recognize nested entities in relational triples. Nested entities are the entities among which there are substring relationships, like U.N. being a substring of U.N. Ambassador. Such entities definitely affect the overall performance on Exact Match mode. Take NYT dataset as an example, there are about 2.5% sentences containing nested entities. Nested entity recognition has been widely studied (Li et al.; Wang et al., 2020a), but most studies on relational triple extraction have not considered it. TPLinker (Wang et al., 2020b) provides a solution to recognize nested entities via a token pair tagging, but it ignores the interaction between entity and relation. For StereoRel model, although not focusing on nested entities, it has not been much affected. The reason is that StereoRel recognizes subjects and objects for each predefined relation separately. In this case, only 0.06% of nested entities in NYT cannot be marked. Anyway, modeling nested entities from the stereoscopic perspective is worth exploring in the future. Relational triple extraction is critical to understanding massive text corpora. However, existing studies face some challenging issues, including information loss, error propagation and ignoring the interaction between entity and relation. In this paper, aiming at simultaneously handling the above issues, we provide a revealing insight into relational triple extraction from a stereoscopic perspective, which rationalizes the occurrence of these issues and exposes the shortcomings of existing methods. Further, we propose a novel model leveraging three decoders to respectively extract subjects, objects and their correspondences for each predefined relation. Extensive experiments are conducted on five public datasets, demonstrating that the proposed model outperforms the recent advanced baselines. This work was supported in part by the National Natural Science Foundation of China under Grant 61822601 and 61773050; the Beijing Natural Science Foundation under Grant Z180006; the Open Project Program Foundation of the Key Laboratory of Opto-Electronics Information Processing, Chinese Academy of Sciences(OEIP-O-202004); National Key Research and Development Project No. 2019YFB1405202.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective objective objective objective objective objective abstain other abstain objective abstain abstain objective objective objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method other abstain other abstain other abstain abstain other other other abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain other