Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Background: Deep learning models are typically trained using stochastic gradient descent or one of its variants. These methods update the weights using their gradient, estimated from a small fraction of the training data. It has been observed that when using large batch sizes there is a persistent degradation in generalization performance -known as the "generalization gap" phenomenon. Identifying the origin of this gap and closing it had remained an open problem. Contributions: We examine the initial high learning rate training phase. We find that the weight distance from its initialization grows logarithmically with the number of weight updates. We therefore propose a "random walk on a random landscape" statistical model which is known to exhibit similar "ultra-slow" diffusion behavior. Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used. We further investigate different techniques to train models in the large-batch regime and present a novel algorithm named "Ghost Batch Normalization" which enables significant decrease in the generalization gap without increasing the number of updates. To validate our findings we conduct several additional experiments on MNIST, CIFAR-10, CIFAR-100 and ImageNet. Finally, we reassess common practices and beliefs concerning training of deep models and suggest they may not be optimal to achieve good generalization.
Summary of the paper and some comments:
This paper investigates the reasons for generalization gap, namely the detrimental effect of large mini-batches on the generalization performance of the neural networks. The paper argues that the reason why the generalization performance is worse when the minibatch size is smaller is due to the less number of steps that the model is trained for. They also discuss the relationship between their method and recent works on flat-minima for generalization. The main message of this paper is that there is no fundamental issue with SGD using large minibatches, and it is possible to make SGD + large minibatches to generalize well as well. The authors propose three different methods to overcome these issues,
1) Ghost BN: They have proposed to use smaller minibatch sizes during the training to compute the population statistics such as mean and the standard deviation. During the test time, they use the full-batch statistics. I think computing the population statistics over the small minibatches, probably have some sort of regularization effect coming from the noise in the estimates. I feel like this is not very well justified in the paper.
2) Learning rate tuning, they propose to rescale the learning rate of the large mini-batches proportionally to the size of the minibatch. This intuitively makes sense, but authors should discuss more and justify it in a better way.
3) Regime adaptation, namely trying to use the same number of iterations as the model being trained. This is justified in the first few sections of the paper.
Firstly, the paper at some points is very verbose and feel like uses the valuable space to explain very simple concepts, e.g. the beginning of section 3 and lacks to provide enough justifications for their approach.
The paper discusses mainly the generalization of SGD with large minibatches, but there is also an aspect of convergence of SGD as well. Because the convergence results of SGD also depends on the minibatch size as well, please see [1]. I would like to see a discussion of this as well.
In my opinion, anyone that spend on monitoring the learning process of the neural networks would notice easily notice that the learning curves usually have a logarithmic behavior.  The relationship to the random-walk theory is interesting, but the relationship that is established in this theory is very weak, it is only obtained by approximating an approximation.
Equation 4 is interesting, for instance, if we initialize w_0 to be all zero, it implies that the norm of the w_t will grow logarithmically with respect to the number of iterations. This is also an interesting property, in my opinion.
Experiments are very difficult to interpret, in particular when those plots are printed on the paper.
The notation is a bit inconsistent in throughout the paper, some parts of the paper use bracket for the expectation and some other parts don't. Authors should explicitly state what the expectation is taken over.
In Figure 3, can you also consider adding the large minibatch cases without regime adaptation for comparison?
It would be interesting to see the learning curves of the models with respect to different wallclock time as well.
I can not see the AB in Table 1 and 2. The results and the improvements are marginal, sometimes it hurts.  It seems like largest improvement comes from RA.
The paper could be better written. I think the paragraph sections in the abstract are unnecessary. The section 7 could be renamed as "Discussion and Conclusion" and could be simplified.
[1] Li, Mu, et al. "Efficient mini-batch training for stochastic optimization." Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2014.