Massively multi-label prediction/classification problems arise in environments like health-care or biology where it is useful to make very precise predictions. One challenge with massively multi-label problems is that there is often a longtailed frequency distribution for the labels, resulting in few positive examples for the rare labels. We propose a solution to this problem by modifying the output layer of a neural network to create a Bayesian network of sigmoids which takes advantage of ontology relationships between the labels to help share information between the rare and the more common labels. We apply this method to the two massively multi-label tasks of disease prediction (ICD-9 codes) and protein function prediction (Gene Ontology terms) and obtain significant improvements in per-label AUROC and average precision.
The authors propose a new training scheme for training neural networks for multi-label prediction tasks by introducing ontology relationships between labels.
The paper motivates very well by the observations that some labels include very small amount of data points.
However, the authors don’t really investigate why such labels are rarely observed and the experiments don’t include any significance.
Thus overall I don’t think the paper is ready for publishing for ICLR yet.
Below are some more detailed comments:
1) The authors discuss nicely about the intuition to introduce the Bayesian networks in the tasks of disease prediction. Essentially, the probability of assigning the label (leaf node) should be account for the probability of it being observed, namely the prior. Thus it is not surprising that for the rare labels, the proposed method would yield higher precision. However, the experiments don’t really include any significance measurement; especially for such tasks where the number of testing examples with rare labels is small (5~10 positive examples), significance measurement or some forms of hypothesis testing is a must-have in order to draw conclusion about the performance comparison. Answering such significance issue with tests for overfitting would be nice.
2) My other major concern is for the protein function prediction task, the reason of why for certain labels, the number of instances is small, could be due to that a) there don’t exist much biological web-lab evidence, or b) among the population, there indeed only exist small number of proteins associated with such labels. The proposed method can address b) but not necessarily address a).
3) The paper discusses the other results very briefly in Section 5 but doesn’t include any experiment comparison. Thus it is not convincing that the proposed method is making contribution to the field of disease prediction, protein function prediction or even general multi-label prediction. I would suggest to include the comparison with the state of the art methods for each application.
4) Particularly for protein function prediction, another line of studies is to use protein protein interaction networks or other sources such as functional pathways rather than using sequence information alone (ref below). Some discussion would be nice.
Schwikowski, Benno, Peter Uetz, and Stanley Fields. "A network of protein–protein interactions in yeast." Nature biotechnology 18.12 (2000): 1257.
Cao, Mengfei, et al. "New directions for diffusion-based network prediction of protein function: incorporating pathways with confidence." Bioinformatics 30.12 (2014): i219-i227.