Open relation extraction (OpenRE) aims to extract novel relation types from open-domain corpora, which plays an important role in completing the relation schemes of knowledge bases (KBs). Most OpenRE methods cast different relation types in isolation without considering their hierarchical dependency. We argue that OpenRE is inherently in close connection with relation hierarchies. To address the bidirectional connections between OpenRE and relation hierarchy, we propose the task of open hierarchical relation extraction and present a novel OHRE framework for the task. To effectively integrate hierarchy information into relation representations for better novel relation extraction, we propose a dynamic hierarchical triplet objective and hierarchical curriculum training paradigm. We also present a top-down hierarchy expansion algorithm to add the extracted relations into existing hierarchies with reasonable interpretability. Comprehensive experiments show that OHRE outperforms state-of-the-art models by a large margin on both relation clustering and hierarchy expansion. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/OHRE . Open relation extraction (OpenRE) aims to extract novel relations types between entities from open-domain corpora, which plays an important role in completing the relation schemes of knowledge bases (KBs). OpenRE models are mainly categorized into two groups, namely tagging-based and clustering-based methods. Tagging-based methods consider OpenRE as a sequence labeling  indicates equal contribution  Corresponding author: Z.Liu (liuzy@tsinghua.edu.cn) significant person relative father participant winner Training instances  Representation Learning Test instances  OHRE child spouse RelationClustering Novel relations  Hierarchy Expansion Figure 1: The workflow of OHRE framework. task, which extracts relational phrases from sentences (Banko et al., 2007; Cui et al., 2018). In contrast, clustering-based methods aim to cluster relation instances into groups based on their semantic similarities, and regard each cluster as a relation (Yao et al., 2011; Wu et al., 2019). However, most OpenRE models cast different relation types in isolation, without considering their rich hierarchical dependencies. Hierarchical organization of relations has been shown to play a central role in the abstraction and generalization ability of human (Tenenbaum et al., 2011). This hierarchical organization of relations also constitutes the foundation of most modern KBs (Auer et al., 2007; Bollacker et al., 2008). Figure 1 illustrates an example of relation hierarchy in Wikidata (Vrandecic and Krtzsch, 2014). Such relation hierarchies are crucial in establishing the relation schemes of KBs, and could also help users better understand and utilize relations in various downstream tasks. expert knowledge and are time-consuming, given the usually large quantity of relations in existing hierarchy and the rapid emergence of novel relations in open domain corpora. 1 Since the ultimate goal of OpenRE is to automatically establish and maintain relation schemes for KBs, it is desirable to develop OpenRE methods that can directly add the extracted novel relations into the existing incomplete relation hierarchy. Moreover, incorporating the hierarchical information of existing relations can also help OpenRE methods to model their interdependencies. Such refined semantic connections among existing relations can provide transferable guidance to better extract new relations. Given the inherent bidirectional connections between OpenRE and relation hierarchy, in this work, we aim to introduce relation hierarchy information to improve OpenRE performance, and directly add the extracted new relations into the existing hierarchy, which presents unique challenges. We propose a novel framework OHRE to consider relation hierarchy in OpenRE. The key intuition behind our framework is that distance between relations in hierarchy reflects their semantic similarity. Therefore, nearby relations should share similar representations, and vice versa. Figure 1 shows the framework of OHRE, which consists of two components: (1) In relation representation learning , we design a dynamic hierarchical triplet objective to integrate hierarchy information into relation representations. We also present a hierarchical curriculum learning strategy for progressive and robust training. (2) In relation hierarchy expansion , we first cluster instances into new relation prototypes and then conduct a top-down hierarchy expansion algorithm to locate new relations into hierarchy. In this way, OHRE encodes hierarchical information into relation representations, which improves classical OpenRE and further enables hierarchy expansion. To verify the effectiveness of hierarchical information and the proposed framework, we conduct experiments over two evaluations, including the classical relation clustering task and a novel hierarchy expansion task. Experimental results on two real-world datasets show that our framework can bring significant improvements on the two tasks, even with partially available hierarchy from KBs. The main contributions of this work are concluded as follows: (1) To the best of our knowl-1 E.g., the number of relations in Wikidata has grown to more than 8 , 000 in the last 6 years. edge, we are the first to address bidirectional connections between OpenRE and relation hierarchy. We propose a novel open hierarchical relation extraction task, which aims to provide new relations and their hierarchical structures simultaneously. (2) We present a novel OHRE framework for the proposed task, which integrates hierarchical information into relation representations for better relation clustering, and directly expands existing relation hierarchies with a top-down algorithm. (3) Comprehensive experiments on two real-world datasets demonstrate the effectiveness of OHRE on both relation clustering and hierarchy expansion. Open Relation Extraction. Recent years have witnessed an upsurge of interest in open relation extraction (OpenRE) that aims to identify new relations in unsupervised data. Existing OpenRE methods can be divided into tagging-based methods and clustering-based methods. Tagging-based methods seek to extract surface form of relational phrases from text in unsupervised (Banko et al., 2007; Banko and Etzioni, 2008), or supervised paradigms (Angeli et al., 2015; Cui et al., 2018; Stanovsky et al., 2018). However, many relations cannot be explicitly represented as surface forms, and it is hard to align different relational tokens with the same meanings. In contrast, traditional clustering-based OpenRE methods extract rich features of sentences and cluster features into novel relation types (Lin and Pan-tel, 2001; Yao et al., 2011, 2012; Elsahar et al., 2017). Marcheggiani and Titov (2016) propose discrete-state variational autoencoder (VAE) that optimizes a relation classifier by reconstruction signals. Simon et al. (2019) introduce skewness loss to enable stable training of VAE. Hu et al. (2020) learn relation representations and clusters iteratively via self-training. Wu et al. (2019) improve conventional unsupervised clustering-based methods by combining supervised and unsupervised data via siamese networks, and achieve state-of-the-art performance. However, existing OpenRE methods cast different relation types in isolation without considering their rich hierarchical dependencies. Hierarchy Information Exploitation. Well-organized taxonomy and hierarchies can facilitate many downstream tasks. Hierarchical information derived from concept ontologies can reveal semantic similarity (Leacock and Chodorow, 1998; Ponzetto and Strube, 2007), and is widely applied in enhancing classification models (Rousu et al., 2005; Weinberger and Chapelle, 2009) and knowledge representation learning models (Hu et al., 2015; Xie et al., 2016). Similar to concept hierarchy, some recent works try to exploit semantic connections from relation hierarchy. In the field of relation extraction, Han et al. (2018a) propose a hierarchical attention scheme to alleviate the noise in distant supervision. Zhang et al. (2019) leverage implicit hierarchical knowledge from KBs and propose coarse-to-fine grained attention for long-tail relations. However, these methods are designed to identify pre-defined relations, and cannot be applied to OpenRE that aims to discover novel relations in open-domain corpora. We divide the open hierarchical relation extraction problem into two phases: (1) learning relation representations with hierarchical information and (2) clustering and linking novel relations to existing hierarchies. Learning relation representation is fundamental to open hierarchical relation extraction. We encode sentences into relation representations using a relation embedding encoder. We assume existing relations are organized in hierarchies, which is common in most modern KBs. Note that while Figure 1 shows one hierarchy tree, the relation hierarchies may contain multiple trees. To fully utilize hierarchy information, we design a dynamic hierarchical triplet objective that integrates hierarchy information into relation representations, and hierarchical curriculum learning for robust model training. Pairwise virtual adversarial training is also introduced to improve the representation generalization ability. Relation Embedding Encoder. We adopt CNN to encode sentences into relation representations. Following previous works (Zeng et al., 2014), given a sentence s and target entity pair ( e h , e t ) , each word in the sentence is first transformed into input representations by the concatenation of word embedding and position embedding indicating the position of each entity. Then the input representation is fed into a convolutional layer followed by a max-pooling layer and a fully-connected layer to obtain the relation representation v  R d . The relation representation is normalized by L2 norm, i.e., Relation EmbeddingEncoder Curriculum Learning Dynamic Margin ! After obtaining relation representations, we measure the similarity of two relation instances by the Euclidean distance between their representations: Dynamic Hierarchical Triplet Loss. To effectively integrate relation hierarchy information into relation representations, we propose a dynamic hierarchical triplet loss for instance representation learning. Triplet loss is widely used in metric learning that encourages a static margin between different categories for distinguishment (Schroff et al., 2015). We argue that good relation representations should also reflect hierarchical information, where relations with close semantics in hierarchy should share similar representations. As the example shown in Figure 2, r 1 i and r 1 j should be closer than r 2 i and r 2 j in representation space, since r 1 i and r 1 j are close to each other in the relation hierarchy. We design a hierarchical triplet objective with a dynamic margin which is determined by the distance between relations in hierarchy. Specifically, the dynamic margin is conducted over the instances of the relations. As shown in Figure 2, given two relations r i and r j sampled by hierarchical curriculum training strategy (which will be introduced later), we randomly sample two instances (namely anchor instance a and positive instance p ) from r i , and an instance (namely negative instance n ) from r j . The hierarchical triplet objective requires model to distinguish the positive pair ( a , p ) from the negative pair ( a , n ) by a distance margin, which is dynamically determined by the length of the shortest path between r i and r j in the hierarchy as follows: L t = (cid:88) r i ,r j  T max[0 , d ( v a , v p ) +  d l ( r i , r j ) 1 + l ( r i , r j )  d ( v a , v n )] , (3) where  d is a hyperparameter, l ( r i , r j ) is the length of the shortest path between r i and r j in the hierarchy, 2 and T is the curriculum training strategy that will be introduced later. Intuitively, the margin increases with the length of the shortest path in the hierarchy, with a relative emphasis on distinguishing nearby relations. Compared to the static margin in vanilla triplet loss, dynamic hierarchical margin can capture the semantic similarities of relations in the hierarchy, leading to representations that can serve not only novel relation clustering but also effective relation hierarchy expansion. Hierarchical Curriculum Learning. In addition to providing direct supervision for representation learning, relation hierarchy can also be useful in providing signals for robust model training. We propose a hierarchical training paradigm, which is a curriculum learning strategy (Bengio et al., 2009) that enables progressive training. The motivation is intuitive: In the early period of training, we choose relations that are easy to distinguish by the model, and gradually transfer to harder ones. Specifically, we sample two relations from the same layer in hierarchy that share ancestor relations (i.e., the relations come from the same tree and are of the same depth), with a gradual transition from shallow to deep layers with respect to their common ancestor, as shown in Figure 2. The training procedure will lead the model to learn relations from coarse to fine grains, since the length of the shortest path between two relations in hierarchy gradually increases as the relation pair goes deeper. 3 In experiments, we find it beneficial to warm-up the training of OHRE under the hierarchical training paradigm, and then switch to two random relations in the later phase. Pair-wise Virtual Adversarial Training. Neural metric learning models may suffer from the over-fitting problem by learning very complex decision hyperplanes. In our case, the problem is severe since relation hierarchies provide strong supervision to metric learning. To address this issue, we 2 The margin is 1 if two relations come from different trees. design pair-wise virtual adversarial training that smooths the representation space by penalizing sharp changes in the space. Specifically, for each randomly sampled instance pair, we add worst-case perturbations, such that the distance between the relation pairs reaches the maximum changes. We penalize the loss changes as follows: L v = (cid:88) v 1 ,v 2 (cid:107) d ( v 1 , v 2 )  d ( v 1 ,  v 2 ) (cid:107) 22 , (4) where  v is obtained by adding the worst-case noise to v . Pair-wise virtual adversarial training encourages smooth and robust metric space, thus improving the generalization ability of OpenRE models. Unlike previous works that adopt virtual adversarial training in classification problems (Miyato et al., 2017; Wu et al., 2019), our pair-wise virtual adversarial training is based on distance in Euclidean space instead of classification probability distributions. We refer readers to the appendix for more details about the pair-wise virtual adversarial training. The final loss is defined as the addition of dynamic hierarchical triplet loss L t and pair-wise virtual adversarial loss L v : L = L t +  v L v , (5) where  v is a hyperparameter. To expand the existing relation hierarchies, we first cluster novel relations in open-domain corpora based on instance representations, and then learn relation prototypes for both relations in the existing hierarchy and novel relations. Finally, new relations are inserted into the existing relation hierarchy by a novel top-down hierarchy expansion algorithm based on relation prototypes. The hierarchy expansion framework is designed based on two key assumptions: (1) A relation prototype is the aggregation of all instances belonging to itself and descendant relations. (2) A relation prototype has the highest similarity with its parent relation prototype, and a lower similarity with its sibling relation prototypes. The rationale of the assumptions is that the semantics of a relation is typically covered by its ancestors. The assump-tion is also aligned with the intuition in relation representation learning, where a relation exhibits the highest similarity with its parent, due to the minimum shortest path length (i.e., the length is 1 ). algorithm (Blondel et al., 2008). Louvain detects communities in a graph by greedily merging data points to clusters based on modularity optimization, and has proven effective in OpenRE (Wu et al., 2019). We construct a weighted undirected graph of the relation instances in the test set, where the connection weight between two instances is determined by the distance between their representations: w ( v 1 , v 2 ) = max[0 , 1  d ( v 1 , v 2 ))] . In experiments, we observe that clusters containing very few instances are typically noisy outliers and are not proper to be regarded as novel relations, which is consistent with Wu et al. (2019). Therefore, we merge instances in these clusters into their closest clusters, measured by the highest connection weight. Then we learn relation prototypes for both relations in the existing hierarchy and novel relations based on the clusters. We represent each relation prototype with instances, where the prototype of a novel relation consists of all its instances, and the prototype of an existing relation contains all instances from itself and all descendant relations. Top-Down Hierarchy Expansion. After obtaining relation prototypes, we link these extracted relations to existing hierarchy by a novel top-down hierarchy expansion algorithm. Following the aforementioned assumptions, for each novel relation, the algorithm finds its parent with the highest similarity in a top-down paradigm. Specifically, for each novel relation, starting from the existing root relations, we iteratively search the relation with the highest similarity in candidates layer by layer. In each layer, the search candidates are obtained by the child relations of the search result in the previous layer. The search process terminates if the similarity decreases compared to the previous layer. The extracted relation will be inserted as the child of the most similar relation, or cast as a singleton if the highest similarity is lower than a threshold, where a higher expansion threshold will lead to more singleton relations. The procedure is shown in Algorithm 1, and we refer readers to experiments for a detailed example. In practice, the similarity between a novel relation and an existing relation is given by the average connection between their prototypes as follows: S ( r i , r j ) = (cid:80) v 1  P i (cid:80) v 2  P j w ( v 1 , v 2 ) | P i |  | P j |  (cid:113) 1 + | P sj | , (7) Algorithm 1 Top-Down Hierarchy Expansion Require: r : A novel relation Require:  W : Expansion threshold 1: Init search candidates C = root relations of trees 2: Init highest similarity in previous layer W = 0 3: while C not empty do 4: Search relation  c = arg max c  CS ( r, c ) 5: if S ( r,  c ) > W then 6: // Move to the next layer 7: Update highest similarity W = S ( r,  c ) 8: Update search candidates C = children of  c 9: else 10: Stop searching 11: if W   W then 12: Expand r as child of  c 13: else 14: Cast r as singleton relation where r i is a novel relation and r j is an existing relation, P i and P j are the corresponding relation prototypes, and | P sj | refers to the number of all descendant relations of r j . In experiments, we find that relations containing more descendant relations in hierarchy tend to exhibit lower average connections with novel relations, due to the margins between the contained descendant relations. By introducing (cid:113) 1 + | P sj | , we balance the connection strength and encourage the model to explore wider and deeper hierarchies. The reason for expanding hierarchy with a top-down paradigm is threefold: (1) The coarse-to-fine-grained hierarchy expansion procedure is bio-plausible, as suggested by cognitive neuroscience studies (Tenenbaum et al., 2011). (2) The decision making procedure following the existing hierarchy structure is interpretable. (3) It can achieve better efficiency since the unlikely branches are pruned in the early search stage. To verify the effectiveness of hierarchical information and OHRE, we conduct comprehensive experiments on relation clustering and hierarchy expansion on two real-world datasets. We also conduct a detailed analysis of OHRE to provide a better understanding of our framework. We refer readers to the appendix for more implementation details. Following previous works (Wu et al., 2019; Hu et al., 2020), we evaluate our framework on FewRel (Han et al., 2018b) and New York Times Freebase (NYT-FB) dataset (Marcheggiani and Titov, 2016). However, the original random data splits are not suitable to benchmark open hierarchical relation extraction task, since the test sets do not well cover different topologies in relation hierarchy. In the test sets, for a majority of relations, their parent relations are not labeled with sentences in the dataset, making them singleton relations. It is desirable to include more diverse and challenging relations with complex topologies in the test sets. Thus we re-split these two datasets to better approximate and provide benchmarks for real-world needs. Considering applications where only incomplete relation hierarchies are available, we only use partial hierarchy from KBs, removing the hierarchy of relations beyond the train sets. FewRel Hierarchy. FewRel (Han et al., 2018b) is a supervised dataset created from Wikipedia and Wikidata. Following Wu et al. (2019), the train set includes 64 relations where each relation has 700 instances. The development set and test set share 16 relations, and each set has 1 , 600 instances. We exchange relations from the original train and test set to include three relation typologies in test set: (1) single relation without a parent ( 6 relations), (2) relation with a parent in train set ( 8 relations), and (3) relation with a parent in test set ( 2 relations). We call this dataset FewRel Hierarchy. NYT-FB Hierarchy. NYT-FB (Marcheggiani and Titov, 2016) is a distantly supervised dataset created from New York Times and Freebase. Following Simon et al. (2019), we filter out sentences with non-binary relations. The train set includes 212 relations with 33 , 992 instances. The development set and test set share 50 relations, and have 3 , 835 and 3 , 858 instances respectively. Each relation in development set and test set has at least 10 instances. We call this dataset NYT-FB Hierarchy. We introduce two task settings and corresponding evaluation metrics. (1) Relation clustering setting is widely adopted in previous OpenRE works to evaluate the ability of clustering novel relations (Marcheggiani and Titov, 2016; Wu et al., 2019). (2) We also design the hierarchy expansion setting to thoroughly test the ability of OpenRE models in expanding existing relation hierarchies. Baselines. We compare OHRE with state-of-the-art OpenRE baselines. (1) Relational Siamese Network augmented with conditional entropy and virtual adversarial training ( RSN-CV ) (Wu et al., 2019) is the state-of-the-art OpenRE method that transfers relational knowledge from labeled data to discover relations in unlabeled data. (2) SelfORE (Hu et al., 2020) utilizes self-training to iteratively learn relation representations and clusters. (3) HAC with re-weighted word embeddings ( RW-HAC ) (Elsahar et al., 2017) is the state-of-the-art rich feature-based method. RW-HAC first extracts rich features, such as entity types, then reduces feature dimension via principal component analysis, and finally clusters the features with HAC. (4) Discrete-state variational autoencoder ( VAE ) (Elsa-har et al., 2017) optimizes a relations classifier via reconstruction signals, with rich features including dependency paths and POS tags. Evaluation Metrics. Following Wu et al. (2019); Hu et al. (2020), we adopt instance-level evaluation metrics to evaluate relation clustering, including B 3 (Bagga and Baldwin, 1998), V-measure (Rosen-berg and Hirschberg, 2007) and Adjusted Rand Index (ARI) (Hubert and Arabie, 1985). We refer readers to the appendix for more detailed descriptions about the evaluation metrics. In this setting, models are required to first cluster novel relations, and then further add the extracted relations into the existing hierarchy in train set. Baselines. To the best of our knowledge, there are no existing OpenRE methods designed to directly expand an existing relation hierarchy. We design two strong baselines based on state-of-the-art OpenRE architectures. (1) RW-HAC for hierarchy expansion ( RW-HAC-HE ) links each novel relation cluster given by RW-HAC to the existing relation cluster with the global highest the Ward's linkage score. The novel relation will be a singleton if the highest score is less than a threshold. (2) RSN-CV for hierarchy expansion ( RSN-CV-HE ) obtains clusters using RSN-CV, and links them to the hierarchy using our top-down expansion algorithm. Here without confusion, we omit the -HE suffixes in model names in the experiment results. Evaluation Metrics. We adopt two metrics to evaluate on cluster-level (1) how well a predicted cluster matches the golden cluster by matching metric (Larsen and Aone, 1999), and (2) how well Dataset Model B 3 V-measure ARI F1 Prec. the predicted cluster links to the golden position in hierarchy by taxonomy metric (Dellschaft and Staab, 2006). We also report two overall evaluation metrics that consider both relation clustering and hierarchy expansion results. Specifically, we report the arithmetic mean and harmonic mean of matching F1 and taxonomy F1. Main Results. Table 1 shows relation clustering results on two datasets, from which we observe that: (1) OHRE outperforms state-of-the-art models by a large margin, e.g., with 6 . 7% , 4 . 3% , 9 . 6% improvements in B 3 , V-measure, and ARI respectively on FewRel Hierarchy. Compared with unsupervised methods, the performance gap is even greater, e.g., more than 30% in B 3 on FewRel Hierarchy. This shows that OHRE can effectively leverage existing relation hierarchy for better novel relation clustering. (2) The improvements of OHRE are consistent in both supervised FewRel Hierarchy dataset and distantly supervised NYT-FB Hierarchy dataset. This indicates that the representation learning and relation clustering procedure of OHRE is robust to noisy relation labels and long-tail relations in different domains. We note that although our model adopts CNN as the relation encoder, it outperforms SelfORE equipped with BERT (Devlin et al., 2019). We expect it would be beneficial to enhance the relation representations in OHRE with pre-trained language models, and we leave it for future work. Ablation Study. We conduct ablations to investigate the contribution of different components, as shown in Table 2. For fair comparisons, we also ablate virtual adversarial training from RSN-CV (Wu et al., 2019). Experimental results show that all components contribute to the final performance. This shows that hierarchical information from existing relations can provide transferable guidance for novel relation clustering. The performance drops most significantly when removing pair-wise virtual adversarial training, indicating the importance of space smoothing to the generalization of OHRE. Main Results. Table 3 shows the results of hierarchy expansion, from which we observe that: (1) OHRE outperforms strong baselines on hierarchy expansion. Compared to baselines, OHRE achieves higher match F1, which indicates that relations extracted by OHRE can be better aligned with golden relations on cluster-level. Moreover, the advantage in taxonomy F1 shows that OHRE can better add the extracted relations in the existing hierarchy. The reasonable overall result shows the potential of OHRE in real-world open hierarchical relation extraction applications. (2) We also conduct hierarchy expansion experiments with golden novel clusters. However, experiment results show no obvious improvements for all models. Particularly, we note that while RW-HAC and RSN-CV achieve seemingly reasonable performance, they always cast novel relation as a singleton and are unable to add the relation to the right place in hierarchy. 4 4 The proportion of singleton relations is 37 . (b) OHRE first clusters novel relations from open-domain corpora, and learns relation prototypes. (c) OHRE then expands relation hierarchy based on relation prototypes in a top-down paradigm. This is because the inconsistent instance representations within each golden cluster will mislead the expansion procedure on cluster-level, which shows integrating hierarchy information into relation representations is of fundamental importance to hierarchy expansion. Besides, the results also show the necessity of re-splitting FewRel to include more hierarchy topologies in test set for better benchmark. Zoom-in Study. To better understand the performance of models on hierarchy expansion, we divide the relations according to their hierarchy topologies and report the performance on FewRel Hierarchy. Table 4 shows the results on three topologies, including (1) single relations without parents (sgl.), (2) relations with parents in train set (p-trn.), and (3) relations with parents in test set (p-tst.). The results show that although models achieve reasonable performance on clustering in all three topologies, they struggle on hierarchy expansion, especially on relations with parents. In comparison, OHRE can handle some relations with parents in train set. However, there is still ample room for improvement. This shows hierarchy expansion is challenging, and we leave further research for future work. To intuitively show how OHRE expands an existing hierarchy with novel relations from open-domain corpora, we visualize the workflow of OHRE on relation composer , as shown in Figure 3. The average connection score increases as the expansion procedure procedure progress from top to down in hierarchy. The expansion procedure terminates when the connection score decreases. The process is not only better aligned with real-world needs, but also provides better interpretability in decision making. In this work, we make the first attempt to address bidirectional connections between OpenRE and relation hierarchy. In the future, we believe the following directions worth exploring: (1) We use a heuristic method to add new relations into hierarchies based on local similarities between relations. In future, more advanced methods can be designed to model the global interaction between new relations and hierarchy, and learn to effectively add the novel relations. (2) We conduct relation representation learning and hierarchy expansion in a pipeline. In the future, end-to-end models can be developed to jointly optimize these important phases for better open hierarchical relation extraction results. This work is supported by the National Key Research and Development Program of China (No. 2020AAA0106501). Yao is also supported by 2020 Tencent Rhino-Bird Elite Training Program.
abstain abstain abstain objective objective method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain objective method objective abstain objective result objective objective objective objective abstain other other other other other other other other other other other other other other other other other other abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain objective objective abstain method abstain other other