On the Local Hessian in Back-propagation
Back-propagation (BP) is the foundation for successfully training deep neural networks. However, BP sometimes has difficulties in propagating a learning signal deep enough effectively, e.g., the vanishing gradient phenomenon. Meanwhile, BP often works well when combining with "designing tricks" like orthogonal initialization, batch normalization and skip connection. There is no clear understanding on what is essential to the efficiency of BP. In this paper, we take one step towards clarifying this problem. We view BP as a solution of back-matching propagation which minimizes a sequence of back-matching losses each corresponding to one block of the network. We study the Hessian of the local back-matching loss (local Hessian) and connect it to the efficiency of BP. It turns out that those designing tricks facilitate BP by improving the spectrum of local Hessian. In addition, we can utilize the local Hessian to balance the training pace of each block and design new training algorithms. Based on a scalar approximation of local Hessian, we propose a scale-amended SGD algorithm. We apply it to train neural networks with batch normalization, and achieve favorable results over vanilla SGD. This corroborates the importance of local Hessian from another side.
#### Summary  The authors attempt to characterize neural network architectures and training methods conducive to successful training. They propose that backpropagation with respect to a loss function is equivalent to a single step of a "back-matching propagation" procedure in which, after a forward evaluation, we alternately optimize the weights and input activations for each block to minimize a loss for the block's output. The authors propose that architectures and training procedures which improve the condition number of the Hessian of this back-matching loss are more efficient and support this by analytically studying the effects of orthonormal initialization, skip connections, and batch-norm. They offer further evidence for this characterization by designing a blockwise learning-rate scaling method based on an approximation of the backmatching loss and demonstrating an improved learning curve for VGG13 on CIFAR10 and CIFAR100.  #### Quality  The authors assembled a good set of data points to make the case for investigating backmatching propagation procedures, but the strength of this case may be insufficient for a complete work at NIPS. The authors acknowledged that one of the assumptions used for the residual connections case may be too strong and that empirical validation is lacking. Additionally, the scale-amended SGD experiment results appear to rely on individual training runs. For experiments at this scale, it seems reasonable to expect some measure of variance from multiple trials.  #### Clarity  -   If I understood the paper's case correctly, the claim that improving the training efficiency requires or implies a well-conditioned local Hessian is not mathematically strict and rather depends on the collection of observations listed above. If this is the case, the authors should consider softening some assertions in the abstract.  -   I believe equation (12) somewhat overloads F_b; would it be correct to rewrite this as $z_b = F_b(W_b, z_{b-1}) = z_{b-1} = \phi_b(W_b, z_{b-1})$?  -   In Section 4, it's not clear how m_{b, W} and m_{b, z} are computed for Algorithm 2.  #### Originality  Variants of back-matching propagation are explored in prior works but these did not apply this procedure to study the efficiency of vanilla backpropagation.  #### Significance  A general characterization of efficiency for deep learning architectures and training procedures would be very valuable, and the proposal offered here appears to be worth further study.