Hierarchical text classification is a challenging subtask of multi-label classification due to its complex label hierarchy. Existing methods encode text and label hierarchy separately and mix their representations for classification, where the hierarchy remains unchanged for all input text. Instead of modeling them separately, in this work, we propose Hierarchy-guided Contrastive Learning (HGCLR) to directly embed the hierarchy into a text encoder. During training, HGCLR constructs positive samples for input text under the guidance of the label hierarchy. By pulling together the input text and its positive sample, the text encoder can learn to generate the hierarchy-aware text representation independently. Therefore, after training, the HGCLR enhanced text encoder can dispense with the redundant hierarchy. Extensive experiments on three benchmark datasets verify the effectiveness of HGCLR. Hierarchical Text Classification (HTC) aims to categorize text into a set of labels that are organized in a structured hierarchy (Silla and Freitas, 2011). The taxonomic hierarchy is commonly modeled as a tree or a directed acyclic graph, in which each node is a label to be classified. As a subtask of multi-label classification, the key challenge of HTC is how to model the large-scale, imbalanced, and structured label hierarchy (Mao et al., 2019). The existing methods of HTC have variously introduced hierarchical information. Among recent researches, the state-of-the-art models encode text and label hierarchy separately and aggregate two representations before being classified by a mixed feature (Zhou et al., 2020; Deng et al., 2021). As denoted in the left part of Figure 1, their main goal is to sufficiently interact between text and structure to * Corresponding author. achieve a mixed representation (Chen et al., 2021), which is highly useful for classification (Chen et al., 2020a). However, since the label hierarchy remains unchanged for all text inputs, the graph encoder provides exactly the same representation regardless of the input. Therefore, the text representation interacts with constant hierarchy representation and thus the interaction seems redundant and less effective. Alternatively, we attempt to inject the constant hierarchy representation into the text encoder. So that after being fully trained, a hierarchy-aware text representation can be acquired without the constant label feature. As in the right part of Figure 1, instead of modeling text and labels separately, migrating label hierarchy into text encoding may benefit HTC by a proper representation learning method. To this end, we adopt contrastive learning for the hierarchy-aware representation. Contrastive learning, which aims to concentrate positive samples and push apart negative samples, has been considered as effective in constructing meaningful representations (Kim et al., 2021). Previous work on contrastive learning illustrates that it is critical 7109 to building challenging samples (Alzantot et al., 2018; Wang et al., 2021b; Tan et al., 2020; Wu et al., 2020). For multi-label classification, we attempt to construct high-quality positive examples. Existing methods for positive example generation includes data augmentation (Meng et al., 2021; Wu et al., 2020), dropout (Gao et al., 2021), and adversarial attack (Wang et al., 2021b; Pan et al., 2021). These techniques are either unsupervised or task-unspecific: the generation of positive samples has no relation with the HTC task and thus are incompetent to acquire hierarchy-aware representations. As mentioned, we argue that both the ground-truth label as well as the taxonomic hierarchy should be considered for the HTC task. To construct positive samples which are both label-guided and hierarchy-involved, our approach is motivated by a preliminary observation. Notice that when we classify text into a certain category, most words or tokens are not important. For instance, when a paragraph of news report about a lately sports match is classified as basketball, few keywords like NBA or backboard have large impacts while the game result has less influence. So, given a sequence and its labels, a shorten sequence that only keeps few keywords should maintain the labels. In fact, this idea is similar to adversarial attack, which aims to find important tokens which affect classification most (Zhang et al., 2020). The difference is that adversarial attack tries to modify important tokens to fool the model, whereas our approach modifies unimportant tokens to keep the classification result unchanged. Under such observation, we construct positive samples as pairs of input sequences and theirs shorten counterparts, and propose Hierarchy-Guided Contrastive Learning (HGCLR) for HTC. In order to locate keywords under given labels, we directly calculate the attention weight of each token embedding on each label, and tokens with weight above a threshold are considered important to according label. We use a graph encoder to encode label hierarchy and output label features. Unlike previous studies with GCN or GAT, we modify a Graphormer (Ying et al., 2021) as our graph encoder. Graphormer encodes graphs by Transformer blocks and outperforms other graph encoders on several graph-related tasks. It models the graph from multiple dimensions, which can be customized easily for HTC task. We propose Hierarchy-Guided Contrastive Learning (HGCLR) to obtain hierarchy-aware text representation for HTC. To our knowledge, this is the first work that adopts contrastive learning on HTC. For contrastive learning, we construct positive samples by a novel approach guided by label hierarchy. The model employs a modified Graphormer, which is a new state-of-the-art graph encoder. Experiments demonstrate that the proposed model achieves improvements on three datasets. Existing work for HTC could be categorized into local and global approaches based on their ways of treating the label hierarchy (Zhou et al., 2020). Local approaches build classifiers for each node or level while the global ones build only one classifier for the entire graph. Banerjee et al. (2019) builds one classifier per label and transfers parameters of the parent model for child models. Wehrmann et al. (2018) proposes a hybrid model combining local and global optimizations. Shimura et al. (2018) applies CNN to utilize the data in the upper levels to contribute categorization in the lower levels. The early global approaches neglect the hierarchical structure of labels and view the problem as a flat multi-label classification (Johnson and Zhang, 2015). Later on, some work tries to coalesce the label structure by recursive regularization (Gopal and Yang, 2013), reinforcement learning (Mao et al., 2019), capsule network (Peng et al., 2019), and meta-learning (Wu et al., 2019). Although such methods can capture the hierarchical information, recent researches demonstrate that encoding the holistic label structure directly by a structure encoder can further improve performance. Zhou et al. (2020) designs a structure encoder that integrates the label prior hierarchy knowledge to learn label representations. Chen et al. (2020a) embeds word and label hierarchies jointly in the hyperbolic space. Zhang et al. (2021) extracts text features according to different hierarchy levels. Deng et al. (2021) 7110 introduces information maximization to constrain label representation learning. Zhao et al. (2021) designs a self-adaption fusion strategy to extract features from text and label. Chen et al. (2021) views the problem as semantic matching and tries BERT as text encoder. Wang et al. (2021a) proposes a cognitive structure learning model for HTC. Similar to other work, they model text and label separately. Contrastive learning is originally proposed in Computer Vision (CV) as a weak-supervised representation learning method. Works such as MoCo (He et al., 2020) and SimCLR (Chen et al., 2020b) have bridged the gap between self-supervised learning and supervised learning on multiple CV datasets. A key component for applying contrastive learning on NLP is how to build positive pairs (Pan et al., 2021). Data augmentation techniques such as back-translation (Fang et al., 2020), word or span permutation (Wu et al., 2020), and random masking (Meng et al., 2021) can generate pair of data with similar meanings. Gao et al. (2021) uses different dropout masks on the same data to generate positive pairs. Kim et al. (2021) utilizes BERT representation by a fixed copy of BERT. These methods do not rely on downstream tasks while some researchers leverage supervised information for better performance on text classification. Wang et al. (2021b) constructs both positive and negative pairs especially for sentimental classification by word replacement. Pan et al. (2021) proposes to regularize Transformer-based encoders for text classification tasks by FGSM (Goodfellow et al., 2014), an adversarial attack method based on gradient. Though methods above are designed for classification, the construction of positive samples hardly relies on their categories, neglecting the connection and diversity between different labels. For HTC, the taxonomic hierarchy models the relation between labels, which we believe can help positive sample generation. Given a input text x = { x 1 , x 2 , ..., x n } , Hierarchical Text Classification (HTC) aims to predict a subset y of label set Y , where n is the length of the input sequence and k is the size of set Y . The candidate labels y i  Y are predefined and organized as a Directed Acyclic Graph (DAG) G = ( Y, E ) , where node set Y are labels and edge set E denotes their hierarchy. For simplicity, we do not distinguish a label with its node in the hierarchy so that y i is both a label and a node. Since a non-root label of HTC has one and only one father, the taxonomic hierarchy can be converted to a tree-like hierarchy. The subset y corresponds to one or more paths in G : for any non-root label y j  y , a father node (label) of y j is in the subset y . In this section, we will describe the proposed HGCLR in detail. Figure 2 shows the overall architecture of the model. Our approach needs a strong text encoder for hierarchy injection, so we choose BERT (Devlin et al., 2019) as the text encoder. Given an input token sequence: x = { [CLS] , x 1 , x 2 , ..., x n  2 , [SEP] } (1) where [CLS] and [SEP] are two special tokens indicating the beginning and the end of the sequence, the input is fed into BERT. For convenience, we denote the length of the sequence as n . The text encoder outputs hidden representation for each token: H = BERT(x) (2) where H  R n  d h and d h is the hidden size. We use the hidden state of the first token ( [CLS] ) for representing the whole sequence h x = h [CLS] . We model the label hierarchy with a customized Graphormer (Ying et al., 2021). Graphormer models graphs on the base of Transformer layer (Vaswani et al., 2017) with spatial encoding and edge encoding, so it can leverage the most powerful sequential modeling network in the graph domain. We organize the original feature for node y i as the sum of label embedding and its name embedding: f i = label _ emb( y i ) + name _ emb( y i ) . Label embedding is a learnable embedding that takes a label as input and outputs a vector with size d h . Name embedding takes the advantage of the name of the label, which we believe contains fruitful information as a summary of the entire class. We use the average of BERT token embedding of 7111 Push apart BERT Encoder Multi-label Classifier [News, Sports], [News, Music], [Travel] [News, Sports], [News, Music], [Travel] Pull together Input tokens Graph Encoder ... ... ... the label as its name embedding, which also has a size of d h . Unlike previous work which only adopts names on initialization, we share embedding weights across text and labels to make label features more instructive. With all node features stack as a matrix F  R k  d h , a standard self-attention layer can then be used for feature migration. To leverage the structural information, spatial encoding and edge encoding modify the Query-Key product matrix AG in the self-attention layer: A Gij = ( f i WGQ )( f j WGK ) T  d h + c ij + b  ( y i ,y j ) (4) where c ij = 1 D (cid:80) Dn =1 w e n and D =  ( y i , y j ) . The first term in Equation 4 is the standard scale-dot attention, and query and key are projected by WGQ  R d h  d h and WGK  R d h  d h . c ij is the edge encoding and  ( y i , y j ) denotes the distance between two nodes y i and y j . Since the graph is a tree in our problem, for node y i and y j , one and only one path ( e 1 , e 2 , ..., e D ) can be found between them in the underlying graph G (cid:48) so that c ij denotes the edge information between two nodes and w e i  R 1 is a learnable weight for each edge. b  ( y i ,y j ) is the spatial encoding, which measures the connectivity between two nodes. It is a learnable scalar indexed by  ( y i , y j ) . The graph-involved attention weight matrix AG is then followed by Softmax, multiplying with value matrix and residual connection & layer normalization to calculate the self-attention, L = LayerNorm(softmax(A G )V + F) (5) We use L as the label feature for the next step. The Graphormer we use is a variant of the self-attention layer, for more details on the full structure of Graphormer, please refer to the original paper. As mentioned, the goal for the positive sample generation is to keep a fraction of tokens while retaining the labels. Given a token sequence as Equation 1, the token embedding of BERT is defined as: { e 1 , e 2 , ..., e n } = BERT _ emb( x ) (6) The scale-dot attention weight between token embedding and label feature is first calculated to determine the importance of a token on a label, q i = e i WQ , k j = l j WK , A ij = q i k T j  d h (7) The query and key are token embeddings and label features respectively, and WQ  R d h  d h and WK  R d h  d h are two weight matrices. Thus, for a certain x i , its probability of belonging to label y j can be normalized by a Softmax function. Next, given a label y j , we can sample key tokens from that distribution and form a positive sample  x . To make the sampling differentiable, we replace the Softmax function with Gumbel-Softmax (Jang et al., 2016) to simulate the sampling operation: P ij = gumbel _ softmax( A i 1 , A i 2 , ..., A ik ) j (8) 7112 Notice that a token can impact more than one label, so we do not discretize the probability as one-hot vectors in this step. Instead, we keep tokens for positive examples if their probabilities of being sampled exceed a certain threshold  , which can also control the fraction of tokens to be retrained. For multi-label classification, we simply add the probabilities of all ground-truth labels and obtain the probability of a token x i regarding its ground-truth label set y as: P i = (cid:88) j  y P ij (9) Finally, the positive sample  x is constructed as:  x = { x i if P i >  else 0 } (10) where 0 is a special token that has an embedding of all zeros so that key tokens can keep their positions. The select operation is not differentiable, so we implement it differently to make sure the whole model can be trained end-to-end. Details are illustrated in Appendix A. The positive sample is fed to the same BERT as the original one,  H = BERT( x ) (11) and get a sequence representation  h x with the first token before being classified. We assume the positive sample should retain the labels, so we use classification loss of the positive sample as a guidance of the graph encoder and the positive sample generation. Intuitively, given a pair of token sequences and their positive counterpart, their encoded sentence-level representation should be as similar to each other as possible. Meanwhile, examples not from the same pair should be farther away in the representation space. Concretely, with a batch of N hidden state of positive pairs ( h i ,  h i ) , we add a non-linear layer on top of them: c i = W 2 ReLU( W 1 h i )  c i = W 2 ReLU( W 1  h i ) (12) where W 1  R d h  d h , W 2  R d h  d h . For each example, there are 2( N  1) negative pairs, i.e., all the remaining examples in the batch are negative examples. Thus, for a batch of 2 N examples Z = { z  { c i }  {  c i }} , we compute the NT-Xent loss (Chen et al., 2020b) for z m as: L conm =  log exp(sim( z m ,  ( z m )) / ) (cid:80) 2 Ni =1 ,i (cid:54) = m exp(sim( z m , z i ) / ) (13) where sim is the cosine similarity function as sim( u, v ) = u  v/ (cid:107) u (cid:107)(cid:107) v (cid:107) and  is a matching function as:  ( z m ) = (cid:26) c i , if z m =  c i  c i , if z m = c i (14)  is a temperature hyperparameter. Following previous work (Zhou et al., 2020), we flatten the hierarchy for multi-label classification. The hidden feature is fed into a linear layer, and a sigmoid function is used for calculating the probability. The probability of text i on label j is: p ij = sigmoid( W c h i + b c ) j (16) where WC  R k  d h and b c  R k are weights and bias. where y ij is the ground truth. The classification loss of the constructed positive examples  LC can be calculated similarly by Equation 16 and Equation 18 with  h i substituting for h i . The final loss function is the combination of classification loss of original data, classification loss of the constructed positive samples, and the contrastive learning loss: L = LC +  LC + L con (19) where  is a hyperparameter controlling the weight of contrastive loss. During testing, we only use the text encoder for classification and the model degenerates to a BERT encoder with a classification head. Datasets and Evaluation Metrics We experiment on Web-of-Science (WOS) (Kowsari et al., 2017), NYTimes (NYT) (Sandhaus, 2008), and RCV1-V2 (Lewis et al., 2004) datasets for comparison and analysis. WOS contains abstracts of published papers from Web of Science while NYT and RCV1-V2 are both news categorization corpora. We follow the data processing of previous work (Zhou et al., 2020). WOS is for single-path HTC while NYT and RCV1-V2 include multi-path taxonomic labels. The statistic details are illustrated in Table 1. Similar to previous work, We measure the experimental results with Macro-F1 and Micro-F1. Implement Details For text encoder, we use bert-base-uncased from Transformers (Wolf et al., 2020) as the base architecture. Notice that we denote the attention layer in Eq. 4 and Eq. 7 as single-head attentions but they can be extended to multi-head attentions as the original Transformer block. For Graphormer, we set the attention head to 8 and feature size d h to 768 . The batch size is set to 12 . The optimizer is Adam with a learning rate of 3 e  5 . We implement our model in PyTorch and train end-to-end. We train the model with train set and evaluate on development set after every epoch, and stop training if the Macro-F1 does not increase for 6 epochs. The threshold  is set to 0 . 02 on WOS and 0 . 005 on NYT and RCV1-V2. The loss weight  is set to 0 . 1 on WOS and RCV1-V2 and 0 . 3 on NYT. and  are selected by grid search on development set. The temperature of contrastive module is fixed to 1 since we have achieved promising results with this default setting in preliminary experiments. Baselines We select a few recent work as baselines. HiAGM (Zhou et al., 2020), HTCInfoMax (Deng et al., 2021), and HiMatch (Chen et al., 2021) are a branch of work that propose fusion strategies for mixed text-hierarchy representation. HiAGM applies soft attention over text feature and label feature for the mixed feature. HTCInfoMax improves HiAGM by regularizing the label representation with a prior distribution. HiMatch matches text representation with label representation in a joint embedding space and uses joint representation for classification. HiMatch is the state-of-the-art before our work. All approaches except HiMatch adopt TextRCNN (Lai et al., 2015) as text encoder so that we implement them with BERT for a fair comparison. Main results are shown in Table 2. Instead of modeling text and labels separately, our model can make more use of the strong text encoder by migrating hierarchy information directly into BERT encoder. On WOS, the proposed HGCLR can achieve 1.5% and 2.1% improvement on Micro-F1 and Macro-F1 respectively comparing to BERT and is better than HiMatch even if its base model has far better performance. BERT was trained on news corpus so that the base model already has decent performance on NYT and RCV1-V2, outperforming post-pretrain models by a large amount. On NYT, our approach observes a 2.3% boost on Macro-F1 comparing to BERT while sightly increases on Micro-F1 and outperform previous methods on both measurements. On RCV1-V2, all baselines hardly improve Micro-F1 and only influence Macro-F1 comparing to BERT. HTCInfoMax experiences a decrease because its constraint on text representation may contradict with BERT on this dataset. HiMatch behaves extremely well on RCV1-V2 with Macro-F1 as measurement while our approach achieves state-of-the-art on Micro-F1. Besides the potential implement difference on BERT encoder, RCV1-V2 dataset provides no label name, which invalids our name embedding for label representation. Baselines like HiAGM and HiMatch only initialize labl embedding with their names so that this flaw has less impact. We will discuss more on name embedding in next section. The main differences between our work and previous ones are the graph encoder and contrastive learning. To illustrate the effectiveness of these two parts, we test our model with them replaced or removed. We report the results on the develop-7114 Model WOS NYT RCV1-V2 Micro-F1 Macro-F1 Micro-F1 Macro-F1 Micro-F1 Macro-F1 Hierarchy-Aware Models TextRCNN (Zhou et al., 2020) 83.55 76.99 70.83 56.18 81.57 59.25 HiAGM (Zhou et al., 2020) 85.82 80.28 74.97 60.83 83.96 63.35 HTCInfoMax (Deng et al., 2021) 85.58 80.05 -83.51 62.71 HiMatch (Chen et al., 2021) 86.20 80.53 -84.73 64.11 Pretrained Language Models BERT (Our implement) 85.63 79.07 78.24 65.62 85.65 67.02 BERT (Chen et al., 2021) 86.26 80.58 -86.26 67.35 BERT+HiAGM (Our implement) 86.04 80.19 78.64 66.76 85.58 67.93 BERT+HTCInfoMax (Our implement) 86.30 79.97 78.75 67.31 85.53 67.09 BERT+HiMatch (Chen et al., 2021) 86.70 81.06 -86.33 68.66 HGCLR 87.11 81.20 78.86 67.96 86.49 68.31 Table 2: Experimental results of our proposed model on several datasets. ment set of WOS for illustration. We first replace Graphormer with GCN and GAT ( r.p. GCN and r.p. GAT), results are in Table 3. We find that Graphormer outperforms both graph encoders on this task. GAT also involves the attention mechanism but a node can only attend to its neighbors. Graphormer adopts global attention where each node can attend to all others in the graph, which is proven empirically more effective on this task. When the graph encoder is removed entirely (r.m. graph encoder), the results drop significantly, showing the necessity of incorporating graph encoder for HTC task. The model without contrastive loss is similar to a pure data augmentation approach, where positive examples stand as augment data. As the last row of Table 3, on development set, both the positive pair generation strategy and the contrastive learn-(a) ing framework have contributions to the model. Our data generation strategy is effective even without contrastive learning, improving BERT encoder by around 1% on two measurements. Contrastive learning can further boost performance by regularizing text representation. We further analyze the effect of incorporating label hierarchy, the Graphormer, and the positive samples generation strategy in detail. Our approach attempts to incorporating hierarchy into the text representation, which is fed into a linear layer for probabilities as in Equation 16. The weight matrix WC can be viewed as label representations and we plot theirs T-SNE projections under default configuration. Since a label and its father should be classified simultaneously, the representation of a label and its father should be similar. Thus, if the hierarchy is injected into the text repre-7115 Variants of Graphormer Micro-F1 Macro-F1 Base architecture 87.46 81.52 -w/o name embedding 86.40 80.40 -w/o spatial encoding 86.88 80.42 -w/o edge encoding 87.25 80.54 Table 4: Performance with variants of Graphormer on development set of WOS. sentation, labels with the same father should have more similar representation to each other than those with a different father. As illustrated in Figure 3, label representations of BERT are scattered while label representations of our approach are clustered, which demonstrates that our text encoder can learn a hierarchy-aware representation. As for the components of the Graphormer, we validate the utility of name embedding, spatial encoding, and edge encoding. As in Table 4, all three components contribute to embedding the graph. Edge encoding is the least useful among these three components. Edge encoding is supposed to model the edge features provided by the graph, but the hierarchy of HTC has no such information so that the effect of edge encoding is not fully embodied in this task. Name embedding contributes most among components. Previous work only initialize embedding weights with label name but we treat it as a part of input features. As a result, neglecting name embedding observes the largest drop, which may explain the poor performance on RCV1-V2. 5.3.3 Effect of Positive Example Generation To further illustrate the effect of our data generation approach, we compare it with a few generation strategies. Dropout (Gao et al., 2021) uses no pos-(a) A high degree of uncertainty associated with the emission inventory for China tends to degrade the performance of chemical transport models in predicting PM2.5 concentrations especially on a daily basis. In this study a novel machine learning algorithm, Geographically Weighted Gradient Boosting Machine (GW-GBM), was developed by improving GBM through building spatial smoothing kernels to weigh the loss function... Tags: CS, Machine Learning (b) Posterior reversible encephalopathy syndrome (PRES) is a reversible clinical and neuroradiological syndrome which may appear at any age and characterized by headache, altered consciousness, seizures, and cortical blindness... Tags: Medical, Headache Figure 4: Two fragments of the generated positive examples. itive sample generation techniques but contrasts on the randomness of the Dropout function using two identical models. Random masking (Meng et al., 2021) is similar to our approach except the remained tokens are randomly selected. Adversarial attack (Pan et al., 2021) generates positive examples by an attack on gradients. As in Table 5, a duplication of the model as positive examples is effective but performs poorly. Instead of dropping information at neuron level, random masking drops entire tokens and boosts Macro-F1 by over 1%, indicating the necessity of building hard enough contrastive examples. The adversarial attack can build hard-enough samples by gradient ascending and disturbance in the embedding space. But the disturbance is not regularized by hierarchy or labels so that it is less effective since there is no guarantee that the adversarial examples remain the label. Our approach guided the example construction by both the hierarchy and the labels, which accommodates with HTC most and achieves the best performance. In Figure 4, we select two cases to further illustrate the effect of labels on positive samples generation. In the first case, word machine strongly indicates this passage belongs to Machine Learning so that it is kept for positive examples. In the second case, syndrome is related to Medical and PRES occurs several times among Headache . Because of the randomness of sampling, our approach cannot construct an example with all keywords. For instance, learning in case one or headache in case two is omitted in this trial, which adds more difficulties for contrastive examples. In this paper, we present Hierarchy-guided Contrastive Learning (HGCLR) for hierarchy text classification. We adopt contrastive learning for migrating taxonomy hierarchy information into BERT encoding. To this end, we construct positive examples for contrastive learning under the guidance of a graph encoder, which learns label features from taxonomy hierarchy. We modify Graphormer, a state-of-the-art graph encoder, for better graph understanding. Comparing to previous approaches, our approach empirically achieves consistent improvements on two distinct datasets and comparable results on another one. All of the components we designed are proven to be effective. We thank all the anonymous reviewers for their constructive feedback. The work is supported by National Natural Science Foundation of China under Grant No.62036001 and PKU-Baidu Fund (No. 2020BD021).
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method objective abstain abstain abstain abstain result objective method method method abstain abstain objective objective objective objective objective other other other other other other other other other other other other other other other other other other other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain other abstain abstain abstain abstain abstain method method abstain other abstain method method abstain other method abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain method method abstain method abstain abstain method method method abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method method method method result method other other