The Effect of Network Width on the Performance of Large-batch Training
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however it besets the convergence of the algorithm and the generalization performance. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that-for a fixed number of parameters-wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.
It has been wide discussed on how to develop algorithms allow large batches, so that one could train neural networks in a distributed environment. The paper investigates the effect of network width on the performance of large-batch training both theoretically and experimentally. The authors claim that with the same number of parameters, it is more likely to train neural networks using proper large batches easily with a wide network architecture. The theoretical support on 2-layers linear/nonlinear networks and multilayer linear networks is also given.   The paper is well-written and easy to follow. It extends the analysis of Dong el.s' paper about gradient diversity and addresses partly the open question proposed in their works about neural networks. The experiments are well-designed to show the advantage of wide network in many aspects. Plenty of extensive demonstrations (multilayer non-linear or residual networks) are provided and the results are informative. I would therefore vote an acceptance for the paper. However, I have some concerns and I hope it could be addressed   1. There are plenty of unused hyper-parameters in Theorems (c_1, c_2, c_max, c_sup, W^*, W, etc.) and also it's vague to say "with arbitrary constant probability", "sufficient large" or "high probability" in formal stated theorems. If you do want these constants, could be better to have some dependence on it. 2. In the proofs of Theorem 1, probability \delta is used to derive the final bound of B_S(w), while I'm not sure, how could you derive the last equation over line 443? should \delta be bounded? please also show how you handle the negative \Theta item in the nominator.  3. The conclusion of Theorem 2 and 3 shows the ratio of the expectation, and it does not imply a direct bound for the batch threshold B_S(w), so please explain more about the intuition that supports your conjecturing, otherwise the theoretical conclusion might be over-claimed. 4. To measure the convergence performance among different networks architectures and batch-size, a fixed convergence threshold is pre-defined, e.g., MNIST to 96% accuracy. I'm wondering is the value sensitive to your claims? It usually takes more epochs to achieve better accuracy in the final phase of training, considering the probably better expression of deep nets, what would happen if you trying to train MNIST to 100% accuracy? 5. I maybe misunderstanding somewhere, but in figure 3, how could you have same parameters with say (b) K=21, L=1 and K=17, L=10?  Based on the comments above, I would therefore vote a weak accept to the work, but I would like to appreciate it if you could make it a better work.  minors:  1. Line 33, duplicated reference [18]; 2. Line 256, redundant word "this" 3. Missing clarification of order symbols (\Theta and \Omega) in Theorems;