Standard deep neural networks (DNNs) used for classification are trained in an end-to-end fashion for very specific tasks -object recognition, face identification, character recognition, etc. This specificity often leads to overconfident models that generalize poorly to samples that are not from the original training distribution. Moreover, they do not allow to leverage information from heterogeneously annotated data, where for example, labels may be provided with different levels of granularity. Finally, standard DNNs do not produce results with simultaneous different levels of confidence for different levels of detail, they are most commonly an all or nothing approach. To address these challenges, we introduce the problem of nested learning: how to obtain a hierarchical representation of the input such that a coarse label can be extracted first, and sequentially refine this representation to obtain successively refined predictions, all of them with the corresponding confidence. We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks. Looking at the problem of nested learning from an information theory perspective, we design a network topology with two important properties. First, a sequence of low dimensional (nested) feature embeddings are enforced. Then we show how the explicit combination of nested outputs can improve both robustness and finer predictions. Experimental results on CIFAR-10, MNIST, and FASHION-MNIST demonstrate that nested learning outperforms the same network trained in the standard end-to-end fashion. Since the network can be naturally trained with mixed data labeled at different levels of nested details, we also study what is the most efficient way of annotating data, when a fixed training budget is given and the cost of labels increases with the levels in the nested hierarchy.
I read the author response, thank you for responding to my questions.
Original review:
This paper presents a hierarchical learning approach that trains neural network classifiers on a known hierarchy of labels.  The experiments show that the approach makes a network more robust to distortion compared to standard end-to-end learning.  The approach in this paper and its formalization of the task are interesting, but the significance of the techniques is somewhat unclear and the experiments could be more thorough in terms of the baselines and data sets considered.
From the related work section, the relationship between this paper and the previous work is somewhat complicated -- it is hard for a reader not deeply familiar with these previous works to understand the unique contribution made in the submission, and assess why it is significant.  For example, in the last paragraph of that section, many of the distinctions drawn between the submitted work and previous methods seem minor (including a confidence measure for a certain prediction, for example) or seemingly subjective (about whether an operation with a previous method was “natural” or could be done “transparently”).  Making crisper, less ambiguous distinctions between this work and previous work would help.
Likewise, the experimental results here do not compare against any of the hierarchical learning approaches discussed in the related work section.  The results show that the paper’s approach is more robust to distortion compared to standard end-to-end learning.  However, I was unclear on why it was not appropriate to compare against the other hierarchical learning methods from previous work.  Also, if the claim is improved robustness, I feel that evaluating against adversarial training (Madry et al., ICLR 2017) or similar approaches is necessary to understand the practical relevance of these improvements.
Finally, experiments that consider larger hierarchies (here, the number of target classes tends to be small, CIFAR-10 and MNIST each have ten classes, meaning the hierarchies are not very rich) would help illustrate the potential power of the techniques.
Minor
I didn’t understand the following statement, and given that it’s a fairly bold claim I would rephrase it or explain it better in the paper body rather than referring the reader to the appendix:
“A standard DNN unknowingly uses low quality data also to train higher layers, even if there is no high level information in the data.”
I don’t understand what the right arrow operator on the top of page 5 means.  From earlier statements it seems that I(f_i(X), Y_i) -> I(X, Y_i) means that the left quantity is approximately equal to the right.  But I’m not sure why to use an arrow for that rather than an \approx symbol.  The arrow would seem to imply that the left approaches the right in the limit, but if that is what you mean you should tell us in the limit of what.  Later the arrow is used to represent links in a Markov chain, adding further confusion for me.
I think it would be helpful if before Equation 1, you mentioned this holds for strictly nested Y_i’s (since earlier in the paper, Y_i referred to more general things).
 
I assume the ECE is computed over held-out validation data (i.e., not training data)?  The paper should say this.
Page 7: lineal -> linear