Hierarchical text classification is an important yet challenging task due to the complex structure of the label hierarchy. Existing methods ignore the semantic relationship between text and labels, so they cannot make full use of the hierarchical information. To this end, we formulate the text-label semantics relationship as a semantic matching problem and thus propose a hierarchy-aware label semantics matching network (HiMatch). First, we project text semantics and label semantics into a joint embedding space. We then introduce a joint embedding loss and a matching learning loss to model the matching relationship between the text semantics and the label semantics. Our model captures the text-label semantics matching relationship among coarse-grained labels and fine-grained labels in a hierarchy-aware manner. The experimental results on various benchmark datasets verify that our model achieves state-of-the-art results. Hierarchical text classification (HTC) is widely used in Natural Language Processing (NLP), such as news categorization (Lewis et al., 2004) and sci-entific paper classification (Kowsari et al., 2017). HTC is a particular multi-label text classification problem, which introduces hierarchies to organize label structure. As depicted in Figure 1, HTC models predict multiple labels in a given label hierarchy, which generally construct one or multiple paths from coarse-grained labels to fine-grained labels in a top-down manner (Aixin Sun and Ee-Peng Lim, 2001). Generally speaking, fine-grained labels are the most appropriate labels for describing the input text. Coarse-grained labels are generally the parent nodes of coarseor fine-grained labels, expressing a more general concept. The key challenges of  *Corresponding author HTC are to model the large-scale, imbalanced, and structured label hierarchy (Mao et al., 2019). Existing work in HTC has introduced various methods to use hierarchical information in a holistic way. To capture the holistic label correlation features, some researchers proposed a hierarchy-aware global model to exploit the prior probability of label dependencies through Graph Convolution Networks (GCN) and TreeLSTM (Zhou et al., 2020). Some researchers also introduced more label correlation features such as label semantic similarity and label co-occurrence (Lu et al., 2020). They followed the traditional way to transform HTC into multiple binary classifiers for every label (Furnkranz et al., 2008). However, they ignored the interaction between text semantics and label semantics (Furnkranz et al., 2008; Wang et al., 2019), which is highly useful for classification (Chen et al., 2020). Hence, their models may not be sufficient to model complex label dependencies and provide comparable text-label classification scores (Wang et al., 2019). A natural strategy for modeling the interaction between text semantics and label semantics is to introduce a text-label joint embedding by label attention (Xiao et al., 2019) or autoencoders (Yeh et al., 2017). Label attention-based methods adopted a self-attention mechanism to identify label-specific information (Xiao et al., 2019). Autoencoder-based methods extended the vanilla Canonical Correlated Autoencoder (Yeh et al., 2017) to a ranking-based autoencoder architecture to produce comparable text-label scores (Wang et al., 2019). However, these methods assume all the labels are independent without fully considering the correlation between coarse-grained labels and fine-grained labels, which cannot be simply transferred to HTC models (Zhou et al., 2020). In this paper, we formulate the interaction between text and label as a semantic matching problem and propose a Hi erarchy-aware Label Semantics Match ing Network (HiMatch). The principal idea is that the text representations should be semantically similar to the target label representations (especially fine-grained labels), while they should be semantically far away from the incorrect label representations. First, we adopt a text encoder and a label encoder (shown in Figure 2) to extract textual semantics and label semantics, respectively. Second, inspired by the methods of learning common embeddings (Wang et al., 2019), we project both textual semantics and label semantics into a text-label joint embedding space where correlations between text and labels are exploited. In this joint embedding space, we introduce a joint embedding loss between text semantics and target label semantics to learn a text-label joint embedding. After that, we apply a matching learning loss to capture text-label matching relationships in a hierarchy-aware manner. In this way, the fine-grained labels are semantically closest to the text semantics, followed by the coarse-grained labels, while the incorrect labels should be semantically far away from the text semantics. Hence, we propose a hierarchy-aware matching learning method to capture different matching relationships through different penalty margins on semantic distances. Finally, we employ the textual representations guided by the joint embedding loss and matching learning loss to perform the hierarchical text classification. 1. By considering the text-label semantics matching relationship, we are the first to formulate HTC as a semantic matching problem rather than merely multiple binary classification tasks. 2. We propose a hierarchy-aware label semantics matching network (HiMatch), in which we introduce a joint embedding loss and a matching learning loss to learn the text-label semantics matching relationship in a hierarchy-aware manner. Hierarchical text classification is a particular multilabel text classification problem, where the classification results are assigned to one or more nodes of a taxonomic hierarchy. Existing state-of-the-art methods focus on encoding hierarchy constraint in a global view such as directed graph and tree structure. Zhou et al. (2020) proposed a hierarchy-aware global model to exploit the prior probability of label dependencies. Lu et al. (2020) introduced three kinds of label knowledge graphs, i.e., taxonomy graph, semantic similarity graph, and co-occurrence graph to benefit hierarchical text classification. They regarded hierarchical text classification as multiple binary classification tasks (Furnkranz et al., 2008). The limitation is that these models did not consider the interaction of label semantics and text semantics. Therefore, they failed to capture complex label dependencies and can not provide comparable text-label classification scores (Wang et al., 2019), which leads to restricted performance (Chen et al., 2020). Hence, it is crucial to exploit the relationship between text and label semantics, and help the model distinguish target labels from incorrect labels in a comparable and hierarchy-aware manner. We perform matching learning in a joint embedding of text and label to solve these problems in this work. To determine the correlation between text and label, researchers proposed various methods to exploit a text-label joint embedding such as (Xiao et al., 2019) or Autoencoder (Yeh et al., 2017). In the field of multi-label text classification, Xiao et al. (2019) proposed a Label-Specific Attention Network (LSAN) to learn a text-label joint embedding by label semantic and document semantic. Wang et al. (2019) extended vanilla Canonical Correlated AutoEncoder (Yeh et al., 2017) to a ranking-based autoencoder architecture to produce comparable label scores. However, they did not fully consider label semantics and holistic label correlation Global debt is set to reach... among fine-grained labels, coarse-grained labels, and incorrect labels. In addition, we can not simply transfer these multi-label classification methods to HTC due to the constraint of hierarchy (Zhou et al., 2020). In this section, we will describe the details about our Hierarchy-aware Label Semantics Matching Network. Figure 2 shows the overall architecture of our proposed model. In the HTC task, given the input sequence x seq = { x 1 , ..., x n } , the model will predict the label y = { y 1 , ..., y k } where n is the number of words and k is the number of label sets. The label with a probability higher than a fixed threshold (0.5) will be regarded as the prediction result. The sequence of token embeddings is firstly fed into a bidirectional GRU layer to extract contextual feature H = { h 1 , ..., h n } . Then, CNN layers with top-k max-pooling are adopted for generating key n-gram features T  R k  d cnn where d cnn indicates the output dimension of the CNN layer. Following the previous work (Zhou et al., 2020), we further introduce a hierarchy-aware text feature propagation module to encode label hierarchy information. We define a hierarchy label structure as a directed graph G = (cid:16) V t ,  E ,  E (cid:17) , where V t indicates the set of hierarchy structure nodes. E are built from the top-down hierarchy paths representing the prior statistical probability from parent nodes to children nodes. E are built from the bottom-up hierarchy paths representing the connection relationship from children nodes to parent nodes. The feature size of graph adjacency matrix  E and  E is  R k  k , where k is the number of label sets. Text feature propagation module firstly projects text features T to node inputs V t by a linear transformation W proj  R k  d cnn  d t , where d t represents the hierarchy structure node dimension from text feature. Then a Graph Convolution Network (GCN) is adopted to explicitly combine text semantics with prior hierarchical information  E and  E : S t =  (cid:16)  E  V t  W g 1 +  E  V t  W g 2 (cid:17) (1) where  is the activation function ReLU . W g 1 , W g 2  R d t  d t are the weight matrix of GCN. S t is the text representation aware of prior hierarchy paths. In the HTC task, the hierarchical label structure can be regarded as a directed graph G = (cid:16) V l ,  E ,  E (cid:17) where V l indicates the set of hierarchy structure nodes with label representation. The graph G in label encoder shares the same structure  E and  E with the graph in text encoder. Given the total label set y = { y 1 , ..., y k } as input, we create label embeddings V l  R d l by averaging of pre-trained label embeddings first. Then GCN could be utilized as label encoder: S l =  (cid:16)  E  V l  W g 3 +  E  V l  W g 4 (cid:17) (2) where  is the activation function ReLU . W g 3 , W g 4  R d l  d l are the weight matrix of GCN. S l is the label representation aware of prior hierarchy paths. It must be noted that the weight matrix and input representation of the label encoder are different with those in the text encoder. In this section, we will introduce the methods of learning a text-label joint embedding and hierarchy-aware matching relationship. For joint embedding learning, firstly, we project text semantics S t and label semantics S l into a common latent space as follows:  t = FFN t ( S t ) , (3)  l = FFN l ( S l ) (4) where FFN t and FFN l are independent two-layer feedforward neural networks. t ,  l  R d  represent text semantics and label semantics in joint embedding space, respectively. d  indicates the dimension of joint embedding. In order to align the two independent semantic representations in the latent space, we employ the mean squared loss between text semantics and target labels semantics: L joint = (cid:88) p  P ( y ) (cid:13)(cid:13)  t   pl (cid:13)(cid:13) 2 2 (5) where P ( y ) is target label sets. L joint aims to minimize the common embedding loss between input text and target labels. Based on the text-label joint embedding loss, the model only captures the correlations between text semantics and target labels semantics, while correlations among different granular labels are ignored. In the HTC task, it is expected that the matching relationship between text semantics and fine-grained labels should be the closest, followed by coarse-grained labels. Text semantics and incorrect labels semantics should not be related. Insight of these, we propose a hierarchy-aware matching loss L match to incorporate the correlations among text semantics and different labels semantics. L match aims to penalize the small semantic distance between text semantics and incorrect labels semantics with a margin  : L match = max (cid:0) 0 , D (cid:0)  t ,  pl (cid:1)  D ( t ,  nl ) +  (cid:1) (6) where  pl represents target labels semantics and  nl represents incorrect labels semantics. We use L2-normalized euclidean distance for metric D and  is a margin constant for margin-based triplet loss. We take the average of all the losses between every label pairs as the margin loss. Hierarchy-aware Margin Due to the large label sets in the HTC task, it is time-consuming to calculate every label's matching loss. Therefore, we propose hierarchy-aware sampling to alleviate the problem. Specifically, we sample all parent labels (coarse-grained labels), one sibling label, and one random incorrect label for every fine-grained label to obtain its negative label sets n  N ( y ) . It is also unreasonable to assign the same margin for different label pairs since the label semantics similarity is quite different in a large structured label hierarchy. Our basic idea is that the semantics relationship should be closer if two labels are closer in the hierarchical structure. Firstly, the text semantics should match fine-grained labels the most, which is exploited in joint embedding learning. Then we regard the pair with the smallest semantic distance ( d 1 ) as a positive pair and regard other text-label matching pairs as negative pairs. As depicted in the schema figure 3, compared with the positive pair, the semantics matching distance between text and coarse-grained target labels ( d 2 ) should be larger. The incorrect sibling labels have a certain semantic relationship with the target labels. Hence, the semantics matching distance between text and the incorrect sibling labels of fine-grained labels ( d 3 ) should be further larger, while the semantics matching distance between text and other incorrect labels ( d 4 ) should be the largest. We introduce hierarchy-aware penalty margins  1 ,  2 ,  3 ,  4 to model the comparable relationship. The penalty margin is smaller if we expect the semantic matching distance to be smaller. We neglect  1 because the matching relationships between text semantics and fine-grained labels are exploited in joint embedding learning. 2 ,  3 ,  4 are penalty margins compared with the matching relationships between text semantics and fine-grained labels semantics. We introduce two hyperparameters ,  to measure different matching relationships of  :  2 =  ;  3 =  ;  4 =  (7) where 0 <  <  < 1 . The proposed loss captures the relative semantics similarity rankings among target labels and incorrect labels in a hierarchy-aware manner. We find that it is easier to overfit for classification learning if we perform classification learning in the text-label joint embedding directly. Hence, we use the text semantics representation S t guided by joint embedding loss and matching learning loss to perform classification learning. S t is fed into a fully connected layer to get the label probability  y for prediction. The overall objective function includes a cross-entropy category loss, joint embedding loss and hierarchy-aware matching loss: L = L cls ( y,  y ) +  1 L joint +  2 L match (8) where y and  y are the ground-truth label and output probability, respectively. 1 ,  2 are the hyperparameters for balancing the joint embedding loss and Dataset | L | Depth Avg ( | L i | ) Train V al Test RCV1-V2 103 4 3.24 20833 2316 781265 WOS 141 2 2 30070 7518 9397 EURLEX-57K 4271 5 5 45000 6000 6000 Table 1: Statistics of three datasets for hierarchical multi-label text classification. matching learning loss. We minimize the above function by gradient descent during training. Datasets To evaluate the effectiveness of our model, we conduct experiments on three widely-studied datasets for hierarchical multi-label text classification. Statistics of these datasets are listed in Table 1. RCV1-V2 (Lewis et al., 2004) is a news categorization corpora, and WOS (Kowsari et al., 2017) includes abstracts of published papers from Web of Science. EURLEX57K is a large hierarchical multi-label text classification (LMTC) dataset that contains 57k English EU legislative documents, and is tagged with about 4.3k labels from the European Vocabulary (Chalkidis et al., 2019). The label sets are split into zero-shot labels, few-shot labels, and frequent labels. Few-shot labels are labels whose frequencies in the training set are less than or equal to 50. Frequent labels are labels whose frequencies in the training set are more than 50. The label setting is the same as previous work (Lu et al., 2020). In EURLEX57K, the corpora are only tagged with fine-grained labels, and the parent labels of fine-grained labels are not tagged as the target labels. Evaluation Metric On RCV1-V2 and WOS datasets, we measure the experimental results by Micro-F1 and Macro-F1. Micro-F1 takes the overall precision and recall of all the instances into account, while Macro-F1 equals the average F1-score of labels. We report the results of two ranking metrics on large hierarchical multi-label text classification dataset EURLEX-57K, including Re-call@5 and nDCG@5. The ranking metrics are preferable for EURLEX-57K since it does not introduce a significant bias towards frequent labels (Lu et al., 2020). (Pennington et al., 2014). Then we use a one-layer BiGRU with hidden dimension 100 and used 100 filters with kernel size [2,3,4] to setup the CNNs. The dimension of the text propagation feature and graph convolution weight matrix are both 300. The hidden size of joint embedding is 200. The matching margin  is set to 0.2 on RCV1-V2 and WOS datasets, and set to 0.5 on EURLEX-57K dataset. We set the value of hierarchy-aware penalty hyperparameters  ,  to 0.01 and 0.5, respectively. The loss balancing factor  1 ,  2 are set to 1. For fair comparisons with previous work (Lu et al., 2020; Chalkidis et al., 2019) on EURLEX-57K dataset, firstly, we do not use CNN layer and text feature propagation module. Secondly, to adapt to the zero-shot settings, the prediction is generated by the dot product similarity between text semantics and label semantics. Our model is optimized by Adam with a learning rate of 1e-4. For pretrained language model BERT (Devlin et al., 2018), we use the top-level representation h CLS of BERT's special CLS token to perform classification. To combine our model with BERT, we replace the text encoder of HiMatch with BERT, and the label representations are initiated by pretrained BERT embedding. The batch size is set to 16, and the learning rate is 2e-5. Comparison Models On RCV1-V2 and WOS datasets, we compare our model with three types of strong baselines: 1) Text classification baselines: TextRCNN (Lai et al., 2015), TextRCNN with label attention (TextRCNN-LA) (Zhou et al., 2020), and SGM (Yang et al., 2018). 2) Hierarchy-aware models: HE-AGCRCNN (Peng et al., 2019), HMCN (Mao et al., 2019), Htrans (Banerjee et al., 2019), HiLAP-RL (Mao et al., 2019) which introduced reinforcement learning to simulate the assignment process, HiAGM (Zhou et al., 2020) which exploited the prior probability of label dependecies through Graph Convolution Network and TreeLSTM. 3) Pretrained language model: a more powerful pretrained language model BERT (Devlin et al., 2018) than tradition text classification models when fine-tuned on downstream tasks. On EURLEX-57K dataset, we compare our model with strong baselines with/without zero-shot settings such as BIGRU-ATT, BIGRU-LWAN (Chalkidis et al., 2019) which introduced label-wise attention. The models starting with ZERO make predictions by calculating similarity scores between text and label semantics for zero-shot settings. AGRU-KAMG (Lu et al., 2020) is a state-of-the-art model which introduced various label knowledge. Table 2, 3 and 4 report the performance of our approaches against other methods. HiAGM is an effective baseline on RCV1-V2 and WOS due to the introduction of holistic label information. However, they ignored the semantic relationship between text and labels. Our model achieves the best results by capturing the matching relationships among text and labels in a hierarchy-aware manner, which achieves stronger performances especially on Macro-F1. The improvements show that our model can make better use of structural information to help imbalanced HTC classification. The pretrained language model BERT is an effective method when fine-tuned on downstream tasks. Compared with the results regarding HTC Frequent Few Zero Overall R@5 nDCG@5 R@5 nDCG@5 R@5 nDCG@5 R@5 nDCG@5 BIGRU-ATT (Chalkidis et al., 2019) 0.740 0.813 0.596 0.580 0.051 0.027 0.675 0.789 BIGRU-LWAN (Chalkidis et al., 2019) 0.755 0.819 0.661 0.618 0.029 0.019 0.692 0.796 ZERO-CNN-LWAN (Chalkidis et al., 2019) 0.683 0.745 0.494 0.454 0.321 0.264 0.617 0.717 ZERO-BIGRU-LWAN (Chalkidis et al., 2019) 0.716 0.780 0.560 0.510 0.438 0.345 0.648 0.752 AGRU-KAMG (Lu et al., 2020) 0.731 0.795 0.563 0.518 0.528 0.414 0.661 0.766 HiMatch 0.769 0.830 0.697 0.648 0.399 0.372 0.705 0.807 Table 4: The experimental results comparing to other state-of-the-art models on EURLEX-57K dataset. as multiple binary classifiers, our results show that the full use of structured label hierarchy can bring great improvements to BERT model on RCV1-V2 and WOS datasets. On EURLEX57K dataset, our model achieves the best results on different matrics except for zero-shot labels. The largest improvements come from few-shot labels. AGRU-KAMG achieves the best results on zero-shot labels by fusing various knowledge such as label semantics similarities and label co-occurrence. However, our model performs semantics matching among seen labels based on training corpora, which is not designed for a specific zero-shot learning task. In this section, we investigate to study the independent effect of each component in our proposed model. Firstly, we validate the influence of two proposed losses, and the hierarchy-aware sampling. The results are reported in Table 5. The results show that F1 will decrease with removing joint embedding loss or matching learning loss. Joint embedding loss has a great influence since label semantics matching relies on the joint embedding. Besides, in the hierarchy-aware margin subsection, we perform hierarchy-aware sampling by sampling coarse-grained labels, incorrect sibling labels, and other incorrect labels as negative label sets. When we remove hierarchy-aware sampling and replace it with random sampling, the results will decrease, which shows the effectiveness of hierarchy-aware sampling. To study the influence of the hyperparameters ,  , and  , we conduct seven experiments on RCV1-V2 dataset. The results are reported in Table 6. The first experiment is the best hyperparameters of our model. Then we fine-tune the matching learning margin  in experiments two and three. We Ablation Models Micro Macro TextRCNN 81.57 59.25 HiMatch 84.73 64.11 w/o Joint Embedding Loss 84.49 62.57 w/o Matching Learning Loss 84.46 63.58 w/o Hierarchy-aware Sampling 84.67 63.45 Table 5: Ablation study on RCV1-V2 dataset. find that a proper margin  = 0 . 2 is beneficial for matching learning compared with a large or small margin. Furthermore, we validate the effectiveness of the hierarchy-aware margin. In experiment four, the performance will decrease if we violate the hierarchical structure by setting a large penalty margin for coarse-grained labels, and setting a small penalty margin for incorrect sibling labels. In experiment five, the performance has a relatively larger decrease if we set  = 1 and  = 1 , which ignores hierarchical structure completely. We speculate that the penalty margin that violates the hierarchical structure will affect the results, since the semantics relationship should be closer if the labels are closer in the hierarchical structure. Moreover, we validate the effectiveness of different penalty margins among different granular labels. In experiments six and seven, the results will degrade if we ignore the relationships between coarse-grained target labels and incorrect sibling labels, by setting the same margin for  and . Therefore, it is necessary to set a small penalty margin for coarse-grained target labels, and a larger penalty margin for incorrect sibling labels. We plot the T-SNE projection of the text representations and label representations in the joint embedding in Figure 4. Figure a) is a part of the hierarchical label structure in RCV1-V2. Label C171 and C172 are fine-grained labels, and label C17 is coarse-grained label of C171 and C172. GWELF and E61 are other labels with different semantics with C17, C171 and C172. In Figure b), by introducing joint embedding loss, we can see that the text representations are close to their corresponding label representations. Furthermore, the text representations of labels C171 and C172 are close to the label representation of their coarse-grained label C17. However, the text representations of different labels may overlap, since the matching relationships among different labels are ignored. In Figure c), by introducing both joint embedding loss and matching learning loss, the text representations of different labels are more separable. Other unrelated text representations and label representations such as labels GWELF, E61 are far away from C17, C171, C172. Besides, the text representations of semantically similar labels (C171 and C172) are far away relatively compared with Figure b). The T-SNE visualization shows that our model can capture the semantics relationship among texts, coarse-grained labels, fine-grained labels and unrelated labels. We analyze the performance with different label granularity based on their hierarchical levels. We compute level-based Micro-F1 and Macro-F1 scores of the RCV1-V2 dataset on TextRCNN, HiAGM, and our model in Figure 5. On RCV1-V2 dataset, both the second and third hierarchical levels contain fine-grained labels (leaf nodes). The second level has the largest number of labels and contains confusing labels with similar concepts, so its Micro-F1 is relatively low. Both the second and third levels contain some long-tailed labels, so their Macro-F1 are relatively low. Figure 5 shows that our model achieves a better performance than other models on all levels, especially among deep levels. The results demonstrate that our model has a better ability to capture the hierarchical label semantic, especially on fine-grained labels with a complex hierarchical structure. In this part, we compare the computational complexity between HiAGM and our model. For time complexity, the training time of HiMatch is 1.11 times that of HiAGM with batch size 64. For space complexity during training, HiMatch has 37.4M parameters, while HiAGM has 27.8M. The increase mainly comes from the label encoder with large label sets. However, during testing, the time and space complexity of HiMatch is the same as HiAGM. The reason is that only the classification results are needed, and we can remove the joint embedding. HiMatch achieves new state-of-the-art results, and we believe that the increase of computational complexity is acceptable. Here we present a novel hierarchical text classification model called HiMatch that can capture semantic relationships among texts and labels at different abstraction levels. Instead of treating HTC as multiple binary classification tasks, we consider the text-label semantics matching relationship and formulate it as a semantic matching problem. We learn a joint semantic embedding between text and labels. Finally, we propose a hierarchy-aware matching strategy to model different matching relationships among coarse-grained labels, fine-grained labels and incorrect labels. In future work, we plan to extend our model to the zero-shot learning scenario. We thank the anonymous reviewers for their helpful feedbacks. The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant No. 61502174, and 61872148), the Natural Science Foundation of Guangdong Province (Grant No. 2017A030313355, 2019A1515010768 and 2021A1515011496), the Guangzhou Science and Technology Planning Project (Grant No. 201704030051, and 201902010020), the Key R&D Program of Guangdong Province (No. 2018B010107002) and the Fundamental Research Funds for the Central Universities.
abstain abstain objective objective method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective method method method abstain objective method abstain abstain other other other other other other other other objective other other other other other abstain method method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain method method abstain method method abstain method abstain method abstain abstain abstain method method method abstain method abstain method method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain method method abstain method method method abstain method other abstain method abstain abstain abstain abstain abstain result result abstain other result result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain objective method abstain objective objective other other