Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions of parameters. We hypothesize that in certain cases it is desirable to relax its intrinsic generalization properties and introduce an extension of SGD called deep gradient boosting (DGB). The key idea of DGB is that back-propagated gradients inferred using the chain rule can be viewed as pseudo-residual targets of a gradient boosting problem. Thus at each layer of a neural network the weight update is calculated by solving the corresponding boosting problem using a linear base learner. The resulting weight update formula can also be viewed as a normalization procedure of the data that arrives at each layer during the forward pass. When implemented as a separate input normalization layer (INN) the new architecture shows improved performance on image recognition tasks when compared to the same architecture without normalization layers. As opposed to batch normalization (BN), INN has no learnable parameters however it matches its performance on CIFAR10 and ImageNet classification tasks.
The paper presents a new gradient update rule for neural net weights, which is a result of relaxing the usual SGD to a regularized linear regression between the descent direction and the original gradient. The method is interpreted as some kind of input normalization similar to Batch Norm later in the paper.
Although the method seems to work and the idea is interesting, I’m not entirely convinced by the experimental evidence. There are some technical concerns:
- In section 2.1, I’m only familiar with the MNIST dataset. Although the proposed method achieves slightly better performance than vanilla SGD, it’s pretty weird the accuracy is only 98.xx% as a very simple CNN could easily achieve >99% accuracy on MNIST. This might be due to the simple network architecture used. A more modern architecture would be preferable in those experiments. Also the improvement on CIFAR seems to close to tell.
- The writing seems to need some improvement. It’s sometimes hard to follow the text, and some result tables (Table A1,A2,A3) are in the appendix not the main text. It might be better to reorganize the text and present everything about the method (fast approximation, CNN, interpretation as normalization) together.
- The authors name the proposed method “deep gradient boosting”, but I’m not entirely sure how it is related to gradient boosting. Eq. 3 concerns sum of weak predictors, but how is this related to Eq. 4? The neural net F(x) doesn’t seem to be exactly a sum of weak learners. I don’t quite follow Eq. 5 – There seems to be missing something.
Typos:
- Eq. 5, d L_t instead of d L_T, and should there be a d \phi / d w_ij term?
- Eq. 7, v_t => v on the right-hand side.
- Eq. 12, should be \sum_i=1^B