summaries: This paper consider multi-label text classification problem and propose a cross attention Transformer encoder to model the correlation between latent labels and input text sequence. The hidden states of those latent labels are concatnate as input to layers of MLP for the classification head. The experiment results show marginal gain over the baselines. This paper proposes to implicitly model label correlations in multi-label text classification. Different from previous studies (e.g., tree-based models) that describe label correlations explicitly, this paper appends "latent labels" to the beginning of each document and feeds it into a BERT classifier. These "latent labels" are randomly initialized, and the concatenation of their output is used for classification.  The authors conduct experiments on two benchmark datasets, AAPD and RCV1. Experimental results show that their proposed method outperforms several baselines for multi-label text classification. Ablation studies further show that using "latent labels" is better than using "actual labels" in their framework. This paper addresses the task of multi-label text classification by modeling the label correlations implicitly. Different from the previous works that explicitly model the label correlations, such as the label embedding methods, this paper proposes modeling the label correlations via latent labels. The proposed method outperforms the baselines on two multi-label text classification benchmarks in the reported experimental results. The paper presents a method that uses latent label representations to model label correlations implicitly, for the multi-label text classification (MLTC) task. The method concatenates a set of randomly generated latent labels to input text tokens. Then the method uses this as the input to the BERT model. At last, the contextual encodings of these latent labels are used to generate predictions for the actual labels. The model has been tested and compared against the LACO algorithm [1] that sets the SOTA on the AAPD and RCV1-V2 datasets, and outperforms LACO using Hamming Loss and Micro-F1. Especially, the proposed method has even better performance than the baseline LACO algorithm, on the low-frequency labels and intensive-label samples.  1. Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning
abstractText: Multi-label text classification (MLTC) aims to assign a set of labels to each given document. Unlike single-label text classification methods that often focus on document representation learning, MLTC faces a key challenge of modeling label correlations due to complex label dependencies. Previous state-of-the-art works model label correlations explicitly. It lacks flexibility and is prone to introduce inductive bias that may not always hold, such as label-correlation simplification, sequencing label sets, and label-correlation overload. To address this issue, this paper uses latent label representations to model label correlations implicitly. Specifically, the proposed method concatenates a set of latent labels (instead of actual labels) to the text tokens, inputs them to BERT, then maps the contextual encodings of these latent labels to actual labels cooperatively. The correlations between labels, and between labels and the text are modeled indirectly through these latent-label encodings and their correlations. Such latent and distributed correlation modeling can impose less a priori limits and provide more flexibility. The method is conceptually simple but quite effective. It improves the state-of-the-art results on two widely used benchmark datasets by a large margin. Further experiments demonstrate that its effectiveness lies in label-correlation utilization rather than document representation. Feature study reveals the importance of using latent label embeddings. It also reveals that contrary to the other token embeddings, the embeddings of these latent labels are sensitive to tasks; sometimes pretraining them can lead to significant performance loss rather than promotion. This result suggests that they are more related to task information (i.e., the actual labels) than the other tokens. 1
references.authors: Ashutosh Adhikari Achyudh Ram Raphael Tang Jimmy Lin
references.title: Docbert: Bert for document classification
references.venue: arXiv preprint arXiv:1904.08398,
references.authors: Matthew R Boutell Jiebo Luo Xipeng Shen Christopher M Brown
references.title: Learning multi-label scene classification
references.venue: Pattern recognition,
references.authors: Linkun Cai Yu Song Tao Liu Kunli Zhang
references.title: A hybrid bert model that incorporates label semantics via adjustive attention for multi-label text classification
references.venue: Ieee Access,
references.authors: Guibin Chen Deheng Ye Zhenchang Xing Jieshan Chen Erik Cambria
references.title: Ensemble application of convolutional and recurrent neural networks for multi-label text categorization
references.venue: In 2017 international joint conference on neural networks (IJCNN),
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: arXiv preprint arXiv:1810.04805,
references.authors: Eva Gibaja Sebastián Ventura
references.title: A tutorial on multilabel learning
references.venue: ACM Computing Surveys (CSUR),
references.authors: Siddharth Gopal Yiming Yang
references.title: Multilabel classification with meta-level features
references.venue: In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval,
references.authors: Jeremy Howard Sebastian Ruder
references.title: Universal language model fine-tuning for text classification
references.venue: arXiv preprint arXiv:1801.06146,
references.authors: Yacine Jernite Anna Choromanska David Sontag
references.title: Simultaneous learning of trees and representations for extreme classification and density estimation
references.venue: In International Conference on Machine Learning,
references.authors: Yoon Kim
references.title: Convolutional neural networks for sentence classification
references.venue: In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing,
references.authors: Gakuto Kurata Bing Xiang Bowen Zhou
references.title: Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence
references.venue: In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Siwei Lai Liheng Xu Kang Liu Jun Zhao
references.title: Recurrent convolutional neural networks for text classification
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: David D Lewis Yiming Yang Tony Russell-Rose Fan Li
references.title: Rcv1: A new benchmark collection for text categorization research
references.venue: Journal of machine learning research,
references.authors: Jingzhou Liu Wei-Cheng Chang Yuexin Wu Yiming Yang
references.title: Deep learning for extreme multilabel text classification
references.venue: In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval,
references.authors: Pengfei Liu Xipeng Qiu Xuanjing Huang
references.title: Recurrent neural network for text classification with multi-task learning
references.venue: arXiv preprint arXiv:1605.05101,
references.authors: Weiwei Liu Xiaobo Shen Haobo Wang Ivor W Tsang
references.title: The emerging trends of multi-label learning
references.venue: arXiv preprint arXiv:2011.11197,
references.authors: Yukun Ma Erik Cambria Sa Gao
references.title: Label embedding for zero-shot fine-grained named entity typing
references.venue: In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers,
references.authors: Aditya Krishna Menon Sadeep Jayasumana Ankit Singh Rawat Himanshu Jain Andreas Veit Sanjiv Kumar
references.title: Long-tail learning via logit adjustment
references.venue: arXiv preprint arXiv:2007.07314,
references.authors: Jinseok Nam Eneldo Loza Mencı́a Hyunwoo J Kim Johannes Furnkranz
references.title: Maximizing subset accuracy with recurrent neural networks in multi-label classification
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Mark Palatucci Dean Pomerleau Geoffrey E Hinton Tom M Mitchell
references.title: Zero-shot learning with semantic output codes. In Advances in Neural Information Processing Systems
references.venue: Annual Conference on Neural Information Processing Systems
references.authors: Hao Peng Jianxin Li Yu He Yaopeng Liu Mengjiao Bao Lihong Wang Yangqiu Song Qiang Yang
references.title: Large-scale hierarchical text classification with recursively regularized deep graph-cnn
references.venue: In Proceedings of the 2018 world wide web conference,
references.authors: Yashoteja Prabhu Manik Varma
references.title: Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning
references.venue: In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining,
references.authors: Jesse Read Bernhard Pfahringer Geoff Holmes Eibe Frank
references.title: Classifier chains for multi-label classification
references.venue: Machine learning,
references.authors: Robert E Schapire Yoram Singer
references.title: Boostexter: A boosting-based system for text categorization
references.venue: Machine learning,
references.authors: Gaurav Singh James Thomas Iain J Marshall John Shawe-Taylor Byron C Wallace
references.title: Structured multi-label biomedical text tagging via attentive neural tree decoding
references.venue: arXiv preprint arXiv:1810.01468,
references.authors: Chi Sun Xipeng Qiu Yige Xu Xuanjing Huang
references.title: How to fine-tune bert for text classification
references.venue: In China National Conference on Chinese Computational Linguistics,
references.authors: Che-Ping Tsai Hung-Yi Lee
references.title: Order-free learning alleviating exposure bias in multi-label classification
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Grigorios Tsoumakas Ioannis Katakis
references.title: Multi-label classification: An overview
references.venue: International Journal of Data Warehousing and Mining (IJDWM),
references.authors: Grigorios Tsoumakas Ioannis Katakis Ioannis Vlahavas
references.title: Mining multi-label data. In Data mining and knowledge discovery handbook
references.authors: Oriol Vinyals Samy Bengio Manjunath Kudlur
references.title: Order matters: Sequence to sequence for sets
references.venue: arXiv preprint arXiv:1511.06391,
references.authors: Guoyin Wang Chunyuan Li Wenlin Wang Yizhe Zhang Dinghan Shen Xinyuan Zhang Ricardo Henao Lawrence Carin
references.title: Joint embedding of words and labels for text classification
references.venue: arXiv preprint arXiv:1805.04174,
references.authors: Ran Wang Robert Ridley Weiguang Qu Xinyu Dai
references.title: A novel reasoning mechanism for multilabel text classification
references.venue: Information Processing & Management,
references.authors: Lin Xiao Xin Huang Boli Chen Liping Jing
references.title: Label-specific document representation for multi-label text classification
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Guangxu Xun Kishlay Jha Jianhui Sun Aidong Zhang
references.title: Correlation networks for extreme multi-label text classification
references.venue: In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
references.authors: Pengcheng Yang Xu Sun Wei Li Shuming Ma Wei Wu Houfeng Wang
references.title: Sgm: sequence generation model for multi-label classification
references.venue: arXiv preprint arXiv:1806.04822,
references.authors: Pengcheng Yang Fuli Luo Shuming Ma Junyang Lin Xu Sun
references.title: A deep reinforced sequenceto-set model for multi-label classification
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Zichao Yang Diyi Yang Chris Dyer Xiaodong He Alex Smola Eduard Hovy
references.title: Hierarchical attention networks for document classification
references.venue: In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies,
references.authors: Dani Yogatama Dan Gillick Nevena Lazic
references.title: Embedding methods for fine grained entity type classification
references.venue: In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume
references.authors: Ronghui You Zihan Zhang Ziye Wang Suyang Dai Hiroshi Mamitsuka Shanfeng Zhu
references.title: Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification
references.venue: Advances in Neural Information Processing Systems,
references.authors: Min-Ling Zhang Zhi-Hua Zhou
references.title: A review on multi-label learning algorithms
references.venue: IEEE transactions on knowledge and data engineering,
references.authors: Ximing Zhang Qian-Wen Zhang Zhao Yan Ruifang Liu Yunbo Cao
references.title: Enhancing label correlation feedback in multi-label text classification via multi-task learning
references.venue: arXiv preprint arXiv:2106.03103,
sections.heading: 1 INTRODUCTION
sections.text: Multi-label text classification (MLTC) aims to associate each text sample with a set of labels. It can be applied to a wide spectrum of scenarios, including text categorization (Schapire & Singer, 2000), information retrieval (Gopal & Yang, 2010) and web mining (Zhang & Zhou, 2013; Liu et al., 2020). Unlike single-label text classification, in MLTC a text is usually associated with more than one label. Typically, these labels correlate with each other. Utilizing their correlations effectively can benefit their predictions significantly (Tsoumakas et al., 2009; Gibaja & Ventura, 2015). However, due to the high-order nature of these correlations, it is a challenging task (Zhang & Zhou, 2013).
  There are different ways to deal with this challenge. A relatively earlier line of works models the correlations independently of text context (Prabhu & Varma, 2014; Jernite et al., 2017; Peng et al., 2018; Singh et al., 2018). A representative class in this line is the tree-structure methods. Typically, they build a hierarchical structure for the label space statistically or according to taxonomy. During predictions, they usually utilize the modeled label correlations through some regularization methods (Peng et al., 2018; Singh et al., 2018). Since the structure of labels that heavily limits the correlations is defined in advance, they have introduced a strong inductive bias that suggests that labels correlate with each other following a tree structure unrelated to text context. But in real tasks, the labels and the text often correlate with each other closely in a complicated way. The correlations between
  1The source code is provided in the supplementary file.
  labels can be quite different depending on text contexts. It may not be appropriate to simplify their correlations a priori too much.
  Another representative class of works in this line is the label embedding methods. In computer vision, researchers have used label embeddings to capture label correlations to enhance performances on zero-shot learning and other tasks (Palatucci et al., 2009; Yogatama et al., 2015; Ma et al., 2016). Inspired by them, Wang et al. (2018) introduce joint label embeddings to model label relations and label-text relations to enhance text classification performances. Their work is extended by Xiao et al. (2019). In these works, the correlations between labels are modeled by the similarities between label embeddings; they are learned adaptively. Such correlation modeling is more flexible and expressive compared with the tree structure methods. And they do not only model label correlations but also model label-text relations. However, they still model label correlations independently of context.
  A relatively recent line of works not only embeds labels but also encodes them contextually through attention-based encoding (or encoding-decoding) mechanisms. Such mechanisms allow them to jointly model the correlations between labels, and between labels and the text context with less a priori limits. This line of works can be further divided into three major classes: sequence-tosequence (Nam et al., 2017; Yang et al., 2018), sequence-to-set (Yang et al., 2019; Tsai & Lee, 2020) and joint encoding (Zhang et al., 2021).
  The sequence-to-sequence methods are first in the line. They are based on the well-known sequenceto-sequence framework. Generally, they encode text by deep neural networks and then decode them to targeted labels one by one. They lead to a breakthrough in this area by eliminating the inductive bias that label correlations are independent of text context. But due to the nature of the classic seq2seq framework, they also introduce another inductive bias: there is an order among the labels associated with each text. However, it is difficult to define such orders appropriately. And this issue is critical to these methods (Vinyals et al., 2015).
  To overcome the difficulties in defining appropriate label orders, the sequence-to-set methods (Yang et al., 2019; Tsai & Lee, 2020) are proposed. Different from seq2seq methods that calculate loss each time a label is generated, they use reinforcement learning to delay loss calculation to the time when the whole label sequence is generated. Since the label sequences are constructed adaptively during training by reinforcement learning, it can obtain them more reasonably. However, the labels are essentially a set rather than a sequence. Constructing the label sequence adaptively still regards the labels as a sequence rather than a set. It does not eliminate the label-sequence inductive bias completely.
  Very recently, a new method (Zhang et al., 2021) (it is the most related work to ours) encodes the text and the labels jointly within a unified encoding process using BERT (Devlin et al., 2018). It performs extra tasks on the label encodings to further model label correlations. By doing so, it removes the label-sequence inductive bias and obtains state-of-the-art results. But their work still suffers from the following three issues. (1) Their extra tasks are designed based on the following consideration: in the context of a given text, the co-occurred labels should have stronger mutual correlations. Since unrelated but common labels can co-occur frequently, we think this inductive bias may still be excessive. We call this issue label-correlation overload. (2) Their extra task plcp models label-correlations pairwisely rather than globally; it may also be an oversimplification. (3) Although jointly encoding actual labels and texts has addressed the label-sequence inductive bias elegantly, it can be less effective and even impossible when the number of actual labels is very large. This issue limits its applications heavily.
  All the above methods share a common fundamental characteristic: they model label correlations explicitly. As one may observe from the above discussions, due to the high-order, contextual and non-sequential nature of label correlations, it is difficult to model them explicitly without introducing improper inductive bias. To overcome this difficulty, this paper uses latent label encodings to model actual labels and their correlations implicitly. Specifically, it pads a set of latent labels rather than actual labels in front of the inputs to BERT, and then maps the contextual encodings of these latent labels to actual class labels cooperatively. The label-label and label-text correlations regarding the actual labels are modeled indirectly through these latent-label encodings and their correlations. Since the correlations between these latent-label encodings, and their relations with actual labels and the text context are all trained adaptively, it imposes fewer a priori limits and provides more flexibility.
  Furthermore, by encoding latent labels rather than actual labels, our method can well address the forementioned three issues of the most related work to ours (Zhang et al., 2021): (1) Our labelcorrelations are learned adaptively with few a priori limits, this can avoid label-correlation overload introduced by empirical extra tasks; (2) By extensively utilizing latent-label encodings and their relations in each label’s prediction, we address the oversimplification of pairwise label-correlation modeling. (3) The number of latent-label is an adjustable hyper-parameter that can be adjusted to accommodate specific tasks, usually it is much smaller than the number of actual labels. This feature addresses the issue of massive-label well.
  General-purpose text classification algorithms are not specially designed for MLTC, such as TextCNN (Kim, 2014) and BERT (Devlin et al., 2018). They do not consider label correlations. In a sense, they can also be regarded as modeling label correlations implicitly. We think that the advantage of our algorithm over them in capturing label correlations lies in the following aspects: compared with their latent neurons, our latent label encodings are structural and high-orderly correlated. These features make it more expressive in modeling label correlations, just like the actual label encodings do.
  We summarize the major contributions of this paper as follows:
  (1) To the best of our knowledge, all the previous state-of-the-art works for MLTC regard label correlations as a key challenge. They model label correlations explicitly, which lacks flexibility and hence is prone to suffer from inappropriate inductive bias. Differently, we use latent label encodings to model label correlations implicitly. The difference between our work and the previous state-ofthe-art works in modeling label correlations is fundamental, it provides a new line of thought.
  (2) Our algorithm outperforms the state-of-the-art results on two widely used benchmark datasets by a large margin. Feature study shows that it significantly outperforms using actual label encodings for classification. Error analysis indicates that the advantage of our algorithm lies in label-correlation utilization rather than document representation. The inspiring results demonstrate the great potential of modeling label correlations implicitly.
  (3) Feature study further reveals that contrary to the other token embeddings, sometimes pretraining latent label embeddings can lead to significant performance loss instead of promotion. This suggests that they are more related to task information (i.e., the actual labels) than the other tokens.
sections.heading: 2 METHODOLOGY
sections.text: Given a text T containing m sequential word tokens w0, w1, . . . , wm−1, wi ∈ V(0 ≤ i ≤ m− 1), a MLTC task aims to determine a label set LS = {l0, l1, . . . , lp−1}, lj ∈ L(0 ≤ j ≤ p− 1, |L| = n), m is the length of the text, LS is the set of labels associated with the text, p is the number of labels in LS, V is the vocabulary of word tokens, L is the complete set of all the candidate labels, n is the cardinality of L. Our method to solve this problem is straightforward: it pads k latent labels ll = (ll0, ll1, . . . , llk−1) in front of the input to BERT and maps the encodings of these latent labels to actual labels through a dense neural network. Figure 1 shows its framework. In the following, we describe them in detail.
  Input
  As illustrated in Figure 1, the input is a concatenation of latent labels and text tokens. Specifically, let the latent labels be ll = (ll0, ll1, . . . , llk−1), the tokenized text be (w0, w1, . . . , wm−1), the input then is (ll0, ll1, . . . , llk−1, w0, w1, . . . , wm−1, [SEP]). These latent labels do not exist in the vocabulary of BERT; their initial embeddings are generated randomly. The number k of latent labels is a hyperparameter. Since all these latent labels share the same position embedding 0, they are a set rather than a sequence. Usually, the number of latent labels is much smaller than that of the actual labels. It makes the input to BERT more concise and easier to train compared with using actual label embeddings. Also, the condensed representations of latent labels makes it easier for them to correlate with each other closely.
  BERT within-task further pretraining (optional)
  The setting of the BERT part is the same as traditional BERT. It is well known that in-domain further pretraining BERT can enhance its adaptability to the specific contexts. So, we employ an optional
  within-task further pretraining process that follows the within-task pretraining procedure introduced in (Sun et al., 2019). Since the latent labels are task-targeted, their embeddings are not pretrained during this process as other tokens do. As shown by our experiment, pretraining them can decrease performance significantly (see 4.2). Note that this is only an optional process; our algorithm can still outperform the state-of-the-art results by a large margin without this process.
  Output and Loss
  As illustrated in Figure 1, let the contextual encodings of these latent labels be Hl = [hll0 ,hll1 , . . . ,hllk−1 ]. They are first flattened and then fed to a single layer dense network to obtain the predictions: ŷ = sigmoid(W To flatten(Hl) + bo) (1) Wo ∈ Rkd×n and bo ∈ Rn are the trainable weights and bias of the dense neural network respectively, d is the dimension of the encodings, n is the number of actual labels. For the recent state-of-the-art LACO algorithm (Zhang et al., 2021), its final prediction on each label is determined only by the encoding of the label itself. On the contrary, our prediction on each label is determined by all the encodings of these latent labels cooperatively. The advantage of our method is twofold. Firstly, it utilizes not only the label encodings but also their correlations. Secondly, the encodings of these latent labels are shared among all the predictions; compared with predicting each label separately, this general sharing can enhance robustness.
  Like previous works, we use Binary Cross Entropy as the loss function for the multi-label text classification tasks:
  Lmltc = − n∑
  i=1
  [qi ln pi + (1− qi) ln (1− pi)] (2)
  where pi = P (yi|D) is the probability of label i predicted by the model (yi ∈ L), and qi ∈ {0, 1} indicates whether yi ∈ LS.
sections.heading: 3 EXPERIMENT
sections.heading: 3.1 DATASETS
sections.text: We evaluate our Latent Label Encoding Method (LLEM) on two widely used multi-label text classification datasets: AAPD (Yang et al., 2018) and RCV1-V2 (Lewis et al., 2004). The AAPD (Arxiv
  Academic Paper) dataset includes 55,840 abstracts of papers involving 54 topics in computer science. RCV1-V2 (Reuters Corpus Volume I) consists of 804,414 manually categorized newswire stories involving 103 topics. We followed the same data-splitting as Yang et al. (2018).
sections.heading: 3.2 EXPERIMENT SETTING
sections.text: We implement our model using Pytorch and run it on 1 NVIDIA Tesla V100 (32G). The model is based on the English base-uncased versions of BERT, just as other BERT-based MLTC models do (Cai et al., 2020; Zhang et al., 2021). The BERT-base model can be further within-task pretrained on the training set to enhance its adaptability to specific contexts (optional) (Sun et al., 2019).
  The hyper-parameters of the optional within-task pretraining of the BERT-based model are set according to Sun et al. (2019). Specifically, the batch size is 32, the max sequence-length is 128, the warm-up steps are 10,000, the learning rate is 5e-5, and the train steps are 100,000.
  The training hyper-parameters of our method are tuned by monitoring Micro-F1 score on the validation set. They are set as follows. The number of latent label is 15 and 20 for AAPD and RCV1-V2 respectively, the batch size is 24, the dropout probability is 0.1, and the max sequence-length is 512. We use Adam method, and set β1 = 0.9, β2 = 0.999 respectively. For sequences longer than 512 (including latent labels), we use head+tail truncation proposed by Sun et al. (2019) with trunc medium = 128. Following Howard & Ruder (2018) and Sun et al. (2019), we utilize layer-learning rate decay strategy with decay factor = 0.95. We also use slanted triangular (Howard & Ruder, 2018) with base learning rate set as 5e-5, and the warm-up proportion as 0.1. The total training epochs are set as 100 and 8 for AAPD and RCV1-V2 respectively. We save the best model on the validation set for testing.
sections.heading: 3.3 EVALUATION METRICS
sections.text: For MLTC tasks, a text is associated with more than one label. The objective is not to find such “a” label correctly, but to find them “all” correctly. Therefore, the recall metric is also important in addition to precision. So instead of precision , the comprehensive metrics Micro-F1 and Hamming Loss are the most important metrics for MLTC tasks (Chen et al., 2017; Yang et al., 2018). Micro-F1 is the weighted average of F1 scores of each class label, while Hamming Loss represents the fraction of misclassified instance-label pairs. They are good metrics to evaluate the overall performance of an MLTC algorithm. Macro-F1 is more sensitive to performance on low-frequency labels. Due to the deficiency of data, the performance on low-frequency labels is highly dependent on effective usage of label correlations. Therefore, Macro-F1 is a good tool to analyze the capacity of an algorithm in utilizing label correlations. Besides these metrics, Micro/Macro-Precision and Micro/Macro-Recall are also reported for reference.
sections.heading: 3.4 COMPARING ALGORITHMS
sections.text: We compare our algorithm with a wide spectrum of competitive algorithms, including the wellknown general-purpose text-classification algorithms: Binary Relevance (BR) (Boutell et al., 2004), TextCNN (Kim, 2014) and BERT (Devlin et al., 2018); label embedding algorithms: LEAM (Wang et al., 2018) and LSAN (Xiao et al., 2019); the sequence generation model (SGM) (Yang et al., 2018); the deep reinforced sequence-to-set model (Yang et al., 2019); the OCD framework (Tsai & Lee, 2020); the ML-Reasoner (Wang et al., 2021); and two recent state-of-the-art BERT based algorithms: the hybrid-BERT (Cai et al., 2020) and LACO (+plcp, +clcp) (Zhang et al., 2021). To the best of our knowledge, these algorithms have provided all the previous state-of-the-art results on the two datasets regarding major metrics.
  Among these algorithms, LACO (+plcp, +clcp) is most related to our Latent Label Encoding Method (LLEM). Both our works encode texts and labels jointly and are based on the BERT-base pretraining model. The key differences are as follows: (1) LACO encodes the actual labels in the same space of text tokens to capture label-label and label-text correlations. On the contrary, our method encodes the latent labels rather than the actual labels, the correlations regarding actual labels are modeled indirectly through these latent-label encodings and their relations. (2) In their work, the encodings of labels are used for prediction (and extra tasks) separately; differently, in our work correlations between the latent label encodings are utilized globally and shared by all the
  predictions; (3) Our work has not performed any extra tasks to further capture label correlations as Zhang et al. (2021); instead, our work has performed an optional unsupervised within-task further pretraining.
sections.heading: 4 RESULTS AND ANALYSIS
sections.text: In this section, we provide the experimental results, and then perform an in-depth analysis on them and our method.
sections.heading: 4.1 EXPERIMENTAL RESULTS
sections.text: We list the experimental results of our algorithms and the compared baselines on the two datasets in Table 1. To the best of our knowledge, the results of the compared baselines include all the previous state-of-the-art results on the two datasets regarding major metrics. Our algorithms for comparison are as follows. (1) LLEM; (2) LLEM+wpre: LLEM that have all the token embeddings (except for the latent label embeddings) within-task pretrained.
  From the results, one can observe that our algorithms are quite competitive. It outperforms the previous state-of-the-art results on both datasets by a large margin regarding the two major metrics MicroF1 and Hamming Loss. Specifically, for the AAPD dataset, LLEM+wpre outperforms the previous best Micro-F1 (provided by LACO+plcp) by 1.47%, previous best Hamming Loss (also provided by LACO+plcp) by 4.25%; for the RCV1-V2 dataset, LLEM+wpre outperforms the previous best Micro-F1 (provided by hybrid BERT) by 1.00%, previous best Hamming Loss (also provided by hybrid BERT) by 11.11%. Even without within-task pretraining, our base LLEM still outperforms the corresponding state-of-the-art results on AAPD by 0.93% and 2.42%, the corresponding stateof-the-art results on RCV1-V2 by 1.22% and 11.11%, respectively. This indicates that our method can achieve substantial improvements by modeling label correlations implicitly through latent label encodings.
sections.heading: 4.2 FEATURE STUDY
sections.text: The objective of the feature study is to evaluate whether using latent labels is value-added compared with using actual labels, as well as investigate the effect of pretraining on latent label embeddings. To this end, we compare our algorithm LLEM+wpre with two baselines. The first baseline using actual label embeddings instead of latent label embeddings and predict each label based on its encoding; we name it as Actual Label Encoding Model (ALEM+wpre). The second baseline is LLEM+wpre all, i.e., LLEM that have all the token (including the latent label) embeddings within-task pretrained. Besides the mentioned difference, all the settings for these compared algorithms are the same. We
  compare these algorithms on the two datasets based on the major metrics, i.e., Hamming Loss, Micro-F1 and Macro-F1. The results are listed in Table 2.
  The results show that our algorithm significantly outperforms ALEM+wpre. It indicates that it is value-added by modeling label correlations implicitly. Compared with Micro-F1, the gap between the two algorithms is even bigger on the Macro-F1 metric, this means the advantage of latent label encoding is more prominent for less frequent labels, sugesting that it is more superior in labelcorrelation utilization. The advantage of our algorithm is also more notable on the AAPD dataset. We think capturing label correlations is more difficult for this dataset due to its much smaller size and more professional text type. This makes the capacity of our method in label-correlation utilization more valuable and hence can lead to more prominent results.
  One can also observe that LLEM+wpre outperforms LLEM+wpre all significantly on the more challenging dataset AAPD. We think it is also attributed to the difficulties in capturing labelcorrelations in AAPD. In such difficult cases, the latent labels should relate to actual labels more closely to better capture their correlations and contribute to their predictions. But pretraining the latent label embeddings without knowing the actual labels cannot contribute to this purpose at all; it can even misguide the latent-label embeddings and make the connection between latent labels and actual labels more difficult. This result suggests that compared with the other token embeddings, the latent label embeddings are sensitive to specific tasks; they are more related to task information (i.e., the actual labels) rather than texts in certain cases.
sections.heading: 4.3 ERROR ANALYSIS
sections.text: LACO (and its extensions LACO+plcp, LACO+clcp) is the latest state-of-the-art baseline. It is the most related work to our method. LACO+plcp also has provided the previous best results on AAPD, which is more challenging in label-correlation utilization due to its much smaller size and more professional text type. In this subsection, we further perform two error analyses on AAPD to provide more in-depth comparisons between LLEM+wpre and LACO+plcp on their label-correlation utilization. We use Macro-F1 as the comparing metric since it is more sensitive to label-correlation utilization, as discussed in 3.3. These two error-analyses compare the two algorithms on labels with different frequencies (F), as well as on samples with different number of labels (L). The design intuition of these two analyses are as follows. Due to the deficiency of data, the performance on low frequency labels are always much lower, their performances are highly dependent on effective usage of label correlations (Yang et al., 2019; Menon et al., 2020). And for intensive-label samples, it is difficult to predict all their labels correctly and completely. But the co-existing labels can contribute to each other’s predictions through their correlations. Therefore, an algorithm that can make effective usage of label-correlations should be capable of providing good results on low-frequency labels, as well as providing good results on intensive-label samples.
  The first error analysis compares our LLEM+wpre with LACO+plcp on labels with different frequencies. We divide the labels into four groups according to their frequencies, just as Zhang et al. (2021): the first group: F > 4500; the second group: 1700 < F ≤ 4500; the third group: 870 < F ≤ 1700; and the last group: F ≤ 870. The results are listed in Fig 2(a). As the frequency decreases, the performances of the two algorithms both decrease notably, indicating the difficulty in predicting low-frequency labels. The two algorithms provide similar results on higher-frequency labels (F > 1700), indicating that they have similar capacities in document representation. But for the lower-frequency labels (F ≤ 1700), our algorithm provides notably higher performance, suggest-
  ing that it is superior in label-correlation utilization. Taken together, these results suggest that the advantage of our algorithm lies in label correlation utilization rather than document representation.
  The second error analysis compares LLEM+wpre with LACO+plcp on samples with different numbers of labels. We divide the samples into three groups according to their number of labels (L): the first group: L ≤ 2; the second group: L = 3; and the last group: L ≥ 4. The results are listed in Fig 2(b). As the number of labels (L) increases, both performances decrease notably, indicating that it is difficult to handle intensive-label samples. The two algorithms provide close results on the sparsest-label samples (L ≤ 2). For such sparsest-label samples, it is very close to single-label text classification. This further indicates that the two algorithms have similar capacities in document representation. But for the more intensive-label samples (L ≥ 3), our algorithm again shows significantly higher performance, indicating that it is more effective in label-correlation utilization. To sum up, these results further demonstrate that the advantage of our algorithm lies in label correlation utilization rather than document representation.
sections.heading: 5 RELATED WORKS
sections.text: The research on multi-label text classification follows two directions: enhancing document representation and modeling label correlations.
  Document representation. Document representation is common for all NLP tasks. Basically, the developments in document representation for MLTC are inspired by other areas of NLP. These include RNN based models (Liu et al., 2016), CNN based models (Kurata et al., 2016; Liu et al., 2017), combining CNN and RNN in models (Chen et al., 2017; Lai et al., 2015), attention-mechanism based models (Yang et al., 2016; You et al., 2019; Adhikari et al., 2019), and most recently the powerful pretrained BERT (Devlin et al., 2018) based methods (Cai et al., 2020; Zhang et al., 2021).
  Modeling label-correlations. Different from single-label text classification (SLTC), multi-label text classification (MLTC) faces a key challenge of modeling label correlations. There are two major lines of works to this end. The first line of works model label-correlations separately to document representation. These include the methods transforming multi-label text classification to multi or binary class text classifications (Tsoumakas & Katakis, 2007; Read et al., 2011), the tree-structure methods (Prabhu & Varma, 2014; Jernite et al., 2017; Peng et al., 2018; Singh et al., 2018), the label embedding methods (Wang et al., 2018; Xiao et al., 2019), and the correlation neural network method (Xun et al., 2020). Since labels and the text often correlate with each other complicatedly, it can be over-simplified to separate these two processes. The other line of works integrates labelcorrelation modeling with document representation by encoding labels contextually with attention mechanisms. These methods include sequence-to-sequence (Nam et al., 2017; Yang et al., 2018), sequence-to-set (Yang et al., 2019; Tsai & Lee, 2020), and most recently the joint encoding with multi-tasks (Zhang et al., 2021). Although they have overcame previous over-simplification inductive bias, they suffer from issues such as label-sequence and label-correlation overload.
sections.heading: 6 CONCLUSIONS
sections.text: How to model label correlations effectively is a specific and important challenge to multi-label text classification. Previous state-of-the-art works for MLTC model label correlations explicitly. It lacks flexibility and hence is prone to introduce inductive bias that may not always hold. Differently, this paper proposes a method to model label correlations implicitly through latent label encodings. The difference between our work and previous state-of-the-art works in modeling label-correlations is fundamental; it provides a new line of thought. The method outperforms the state-of-the-art results on two widely used benchmarks by a large margin. Analyses also show that the effectiveness of our method lies in label-correlation utilization rather than document representation learning. The inspiring results demonstrate the great potential of modeling label correlations implicitly.
  The other contributions of this paper include but are not limited to: the concepts of sparse-label (SL) samples and intensive-label (IL) samples for analyzing label-correlation utilization; revealing the singular nature of latent label embeddings compared with the other token embeddings regarding pretraining.
  In future works, we think it is intresting for us and the other researchers to investigate how the latent-label correlate with each other, with the actual labels and the context, and how can they be interpreted. They certainly can shed light on new methods of their design and training.