This paper proposes a novel approach to train deep neural networks by unlocking the layer-wise dependency of backpropagation training. The approach employs additional modules called local critic networks besides the main network model to be trained, which are used to obtain error gradients without complete feedforward and backward propagation processes. We propose a cascaded learning strategy for these local networks. In addition, the approach is also useful from multi-model perspectives, including structural optimization of neural networks, computationally efficient progressive inference, and ensemble classification for performance improvement. Experimental results show the effectiveness of the proposed approach and suggest guidelines for determining appropriate algorithm parameters.
This paper describes a method of training neural networks without update locking. The idea is a small modification on top of Czarnecki et al. Critic training, where instead of using final loss as a critic target, one bootstraps from critics on other layers. In particular, if only one module is present, these two approaches are actually identical. To be more precise, the only difference between these two methods is that (7) in Critic training would change to l(L_i, L_N). As a consequence, method becomes forward unlocked too. It is worth noting, that in the appendix of Czarnecki et al. it is shown that this particular method (critic training) under simple conditions actually "degenerates" to deep supervision (which is forward unlocked too). Consequently unlocking property as such is not a big contribution of the proposed method. Rest of the paper includes following elements:
- empirical evaluation showing improvement over critic training by 0.4% in CIFAR10 and 0.9% in CIFAR100 when using 3 splits.
- expansion on using the model for progressive inference.
Given standard deviation of errors in Table 1. it is not clear how significant these improvements are. How many samples were used to estimate these quantities? It is worth noting, that Critic training was showed to be outperformed by Sobolev Training in the same paper authors cite, but its performance is not reported despite looking like a well defined baseline. In particular, can these two methods be combined? 
I believe that this is an interesting research direction, however paper in its current form seems as a small incremental improvement over sota, and could be significantly improved by for example:
- providing more comprehensive evaluation (including estimating accuracy to lower std errors)
- adding other baseline solutions (such as Sobolev training, cDNI, or deep supervision)
- considering any form of convergence/dynamics analysis of the proposed approach