We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures. Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures. Our method only requires the relatively inexpensive sequential training of T "shallow ResNets". We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l 1 norm bounded weights.
Summary:
This paper considers a learning method for the ResNet using the boosting framework. More precisely, the authors view the structure of the ResNet as a (weighted) sum of base networks (weak hypotheses) and apply the boosting framework. The merit of this approach is to decompose the learning of complex networks to that of small to large networks in a moderate way and it uses less computational costs. The experimental results are good. The authors also show training and generalization error bounds for the proposed approach.
Comments: 
The idea of the paper is natural and interesting. Experimental results are somewhat impressive. However, I am afraid that theoretical results in the paper contain several mistakes and does not hold. The details are below.
I think the proof of Theorem 4.2 is wrong. More precisely, there are several possibly wrong arguments as follows:
- In the proof, \alpha_t+1 is chosen so as to minimize an upper bound of Z_t, while the actual algorithm is chosen to minimize Z_t. The minimizer of Z_t and that of an upper bound are different in general. So, the obtained upper bound does not hold for the training error of the actual algorithm. 
- It is not a mistake, but, there is no explanation why the equality between (27) and (28) holds. Please add an explanation. Indeed, equation (21) matters. 
Also, the statement of Theorem 4.2 looks somewhat cheating: The statement seems to say that it holds for any iteration T and the training error decays exponentially w.r.t. T. However, the parameter T is determined by the parameter gamma, so it is some particular iteration, which might be small and the bound could be large. 
The generalization error bound Corollary 4.3 seems to be wrong, too. More precisely, Lemma 2 of Cortes et al. is OK, but the application of Lemma 2 is not. In particular, the proof does not take into account of the function \sigma. In other words, the proof considers the Rademacher complexity R_m(\calF_t), of the class \calF_t, but, acutually, I think it should consider R_m(\sigma(\calF_t)), where the class \sigma(\calF_t) consists of the composition of functions \sigma and f_t in \calF_t. Talagrand’s lemma (see, e.g., Mohri et al.’ s book: Foundation of Machine Learning) can be used to analyze the complexity of the composite class. But, the resulting bound would depend on the Lipschizness of \sigma in an exponential way. 
The explanation of the generalization ability is not sufficient. While the latter weak hypotheses are complex enough and would have large edges, the complexity of the function class of weak hypotheses grows exponentially w.r.t. the iteration T, which should be mentioned. 
As a summary, the paper contains nice ideas and experimental results are promising, but has non-negligible mistakes in theoretical parts which degrade the contribution of the paper.
Minor Comments:
-In Algorithm 1, \gamma_t is not defined when a while-loop starts. So, the condition of the while-loop cannot be checked.