Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image recognition tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems leads to a CNN that exceeds AlexNet performance on ImageNet. Extending our training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds VGG-11 on ImageNet obtaining 89.8% top-5 single crop. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We conduct a wide range of experiments to study the properties this induces on the intermediate layers.
Summary:
This paper proposes layer wise training of neural networks using classification auxiliary tasks for training each layer. Experiments are presented on CIFAR10 and Imagenet. Accuracies close to end to end training are obtained. 
The layer wise training is repeated for J steps, the auxiliary tasks are trained on top of the shallow one layer (of width M ) with a network  of depth k and width tilde{M}. Layerwise training is done using sgd with momentum, and the learning rate is decayed through epochs. 
Note that the layer wise training is done with large width M than typical end to end networks in use. 
The authors argue and test the hypothesis that auxiliary tasks  encourage the linear separability of CNN features. 
To reduce the size of the learned network the author propose a layer wise compression using the filter removal technique of Molchanov et al .
Reproducibility:
This empirical  work has been investigated for a while with mild success, the authors should make their code available to the community to confirm and reproduce  their findings.  I encourage the authors to make their code available during the review/discussion period. 
Significance of the work:
From reading the paper it is not clear what is the main ingredient that makes this layer wise training  successful, negative results would help in understanding what is important for the accuracy. 
Some more ablation studies and negative results will be insightful and here are few suggestions in that direction:
- Authors claim that they used invertible downsampling as max or average pooling  lead to information loss. Does the layer wise training give worst results with average or max pooling? If so please report those numbers to know what is the implications of this choice of pooling.  
- On the width of the networks, seems it is key for the success of the approach.  What if  you train wider networks with J that is small? (  J=3 for instance but much  wider networks, instead of J=8 now for imagenet.)
- To answer the same question above one needs also to see what are the accuracies for J=8 with thiner networks (smaller M )?
- Would the accuracy  with the layer wise training reach a  plateau if one uses an architecture with J higher than 8? 
- Transferability of the learned features: end to end features are know to be transferable. It would be good to see if this still holds using the network layer wise trained on imagenet for CIFAR10 or other datasets. 
Other Questions:
- Section 3.2 is vague. In Proposition 3.1 and  Proposition 3.2 can you add some text to explain what are the implicitions of the claims? “Thus our training permits to extend some results on shallow CNN to deeper CNNs …” which shallow results ?
- “For k>1 batch normalization was useful “ is this only on the auxiliary problems networks  or you used also batch norm for the layer ?
- The ensemble used is Z=\sum_{j=1}^J 2^j z_j , this uses the network of J layers ,  also the O(J) auxiliary networks  of depth k. Please report the number of parameters for all models (single and ensembles) in Table 1 and Table 2. 
- In the conclusion: “The framework can be extendable to the parallel training …” how would this possible since one needs the output of the first training to do the training of the next layer. can you elaborate on what is meant here?
Minor:
page 2 bottom have competitorsand -> have competitors and 
the non linearity rho in equation 1 and throughout the paper put a bracket for its argument \rho(x) not \rho x
Page 6 , Imagenet paragraph : W —> We
section 4.2 we define linear separability etc… a space is missing before Further 
section 4.3 we report our results .. (Imagenet) a space is missing after ImageNet)
Overall Assessment: 
This is a good paper, making the code available and adding more ablation studies and explanations of width versus depth and the choice of pooling will make the contributions easier to understand.