Maximizing Subset Accuracy with Recurrent Neural Networks in Multi-label Classification
Multi-label classification is the task of predicting a set of labels for a given input instance. Classifier chains are a state-of-the-art method for tackling such problems, which essentially converts this problem into a sequential prediction problem, where the labels are first ordered in an arbitrary fashion, and the task is to predict a sequence of binary values for these labels. In this paper, we replace classifier chains with recurrent neural networks, a sequence-to-sequence prediction algorithm which has recently been successfully applied to sequential prediction tasks in many domains. The key advantage of this approach is that it allows to focus on the prediction of the positive labels only, a much smaller set than the full set of possible labels. Moreover, parameter sharing across all classifiers allows to better exploit information of previous decisions. As both, classifier chains and recurrent neural networks depend on a fixed ordering of the labels, which is typically not part of a multi-label problem specification, we also compare different ways of ordering the label set, and give some recommendations on suitable ordering strategies.
I have been reviewing this paper for another conference, so in my review I mostly repeat my comments sent earlier. Already at that time I was for accepting the paper. It is worth to underline that the paper has been further improved by the authors since then.
The paper considers a problem of solving multi-label classification (MLC) with recurrent neural networks (RNNs). MLC is converted into a sequential prediction problem: the model predicts a sequence of relevant labels. The prediction is made in a sequential manner taking into account all previous labels. This approach is similar to classifier chains, with two key differences: firstly, only single model is used, and its parameters are shared during prediction; secondly, RNN predicts "positive" labels only. In this way, the algorithm can be much more efficient than the standard classifier chains. The authors discuss also the problem of the right ordering of labels to be used during training and testing.
The paper is definitely very investing. The idea of reducing MLC to sequential prediction problem is a natural extension of chaining commonly used for solving MLC under 0/1 loss. I suppose, however, that some similar ideas have been already considered. The authors could, for example, discuss a link between their approach and the learn-to-search paradigm used, for example, in "HC-Search for Multi-Label Prediction: An Empirical Study". I suppose that the algorithm presented in that paper could also be used for predicting sequences of "positive" labels.
Minor comments:
- The authors write that "Our first observation is that PCCs are no longer competitive to the sequence prediction approaches w.r.t. finding the mode of the joint distribution (ACC and maF1)". It seems however that PCC performs better than RNN under the ACC measure. 
- FastXML can be efficiently tuned for macro-F (see, "Extreme F-measure Maximization using Sparse Probability Estimates"). It is not clear, however, how much this tuning will improve the result on BioASQ.
After rebuttal:
I thank the authors for their response.