  Large-scale Multi-label Text Classification (LMTC) has a wide range of
Natural Language Processing (NLP) applications and presents interesting
challenges. First, not all labels are well represented in the training set, due
to the very large label set and the skewed label distributions of LMTC
datasets. Also, label hierarchies and differences in human labelling guidelines
may affect graph-aware annotation proximity. Finally, the label hierarchies are
periodically updated, requiring LMTC models capable of zero-shot
generalization. Current state-of-the-art LMTC models employ Label-Wise
Attention Networks (LWANs), which (1) typically treat LMTC as flat multi-label
classification; (2) may use the label hierarchy to improve zero-shot learning,
although this practice is vastly understudied; and (3) have not been combined
with pre-trained Transformers (e.g. BERT), which have led to state-of-the-art
results in several NLP benchmarks. Here, for the first time, we empirically
evaluate a battery of LMTC methods from vanilla LWANs to hierarchical
classification approaches and transfer learning, on frequent, few, and
zero-shot learning on three datasets from different domains. We show that
hierarchical methods based on Probabilistic Label Trees (PLTs) outperform
LWANs. Furthermore, we show that Transformer-based approaches outperform the
state-of-the-art in two of the datasets, and we propose a new state-of-the-art
method which combines BERT with LWANs. Finally, we propose new models that
leverage the label hierarchy to improve few and zero-shot learning, considering
on each dataset a graph-aware annotation proximity measure that we introduce.

Computation and Language