Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favors flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets.
Summary:  This paper presents many empirical results on the effects of large batch training based on second order analysis. The results focus on the generalization gap and robustness to adversaries under the setting of large batch training. The first major conclusion is that large batch size training tends to get attracted to areas with higher Hessian spectrum, rather than saddle points. Secondly, large batch training drastically reduces model robustness, and robust training favors parameters with smaller Hessian spectrum.  Strength:  This paper presents many intriguing results under the setting of large batch training. Experiments are comprehensive. Despite lacking theoretical support, the results in this paper help us understand large batch training better and develop new theoretical frameworks. This is also a pioneering result on empirically investigating Hessian-based analysis to understand large batch training, especially on its connections to model robustness.  The presentation of the paper is nice, with many nicely drawn figures and tables. The paper is overall well written. Good job!  Empirical studies in this paper are also done very carefully with sufficient details presented. So I am confident that the experimental results can be reproduced by others.  Weakness:  Despite the Hessian based analysis shows some intriguing properties of the networks under investigation, all experiments in this paper use ReLU networks. In fact, the only contributing term in Hessian of the network comes from the softmax layer (Appendix line 388), which does not contain any trainable parameter. Removing the softmax layer, the network still works as a valid classifier (and we pick the class with largest logit value as the prediction result), but the analysis in this paper would fail as the Hessian is 0 almost everywhere. Given this fact, I am in doubt if the results discovered in this paper actually reflects the property of ReLU networks. In fact, the power of the ReLU network comes from the singular points where Hessian does not exists, which provides non-linearity; without these points the network would become a pure linear network.  I would suggest that conducting empirical analysis on networks with twice-differentiable activations, for example, ELU or softplus, to give more convincing conclusions. Especially, softplus function can approximate ReLU in its limit. The authors can change the scaling parameter in softplus to simulate the case where the Hessian is close to the one of ReLU networks, and compare this result to the existing results presented in this draft.  The theoretical contribution of this paper is also weak; the PSD property of Hessian matrices comes from a simple fact the Hessian of softmax function is  PSD.  Overall Evaluation:  I appreciate the comprehensive experiments, detailed experiment setup, and intriguing findings in this paper. Empirical results are interesting, but are not completely sound theoretically (see the weakness about). Overall I am leaning slightly towards an rejection, but accepting it would not be that bad.  Comments after author feedback: I have carefully read the author feedback. I am glad that the authors conduct additional experiments on twice differentiable activation functions and provide some supporting evidence. I believe the results of this paper are basically correct (despite using only ReLU), but I think the main texts should include more experiments using twice differentiable networks to make the conclusions more convincing (and this requires a relatively large change to this paper). I am fine if the AC wants to accept this paper, and I hope the authors can revise the paper accordingly if accepted.