This paper makes two contributions towards understanding how the hyperparameters of stochastic gradient descent affect the final training loss and test accuracy of neural networks. First, we argue that stochastic gradient descent exhibits two regimes with different behaviours; a noise dominated regime which typically arises for small or moderate batch sizes, and a curvature dominated regime which typically arises when the batch size is large. In the noise dominated regime, the optimal learning rate increases as the batch size rises, and the training loss and test accuracy are independent of batch size under a constant epoch budget. In the curvature dominated regime, the optimal learning rate is independent of batch size, and the training loss and test accuracy degrade as the batch size rises. We support these claims with experiments on a range of architectures including ResNets, LSTMs and autoencoders. We always perform a grid search over learning rates at all batch sizes. Second, we demonstrate that small or moderately large batch sizes continue to outperform very large batches on the test set, even when both models are trained for the same number of steps and reach similar training losses. Furthermore, when training Wide-ResNets on CIFAR-10 with a constant batch size of 64, the optimal learning rate to maximize the test accuracy only decays by a factor of 2 when the epoch budget is increased by a factor of 128, while the optimal learning rate to minimize the training loss decays by a factor of 16. These results confirm that the noise in stochastic gradients can introduce beneficial implicit regularization.
The paper attempts to clarify the debate on large-batch neural network training, particularly on the relationship between learning rate, batch sizes and test performance. The authors claim two contributions towards understanding how the hyper-parameters of SGD affect final training and test performance: (1) SGD exhibits two regimes with different behaviours and (2) large-batch training leads to degradation of test performance even with same step budgets.
Overall, the authors did a comprehensive study on large-batch training with the support of extensive experiments. But I'm concerned with the novelty and contributions of this paper. I tend to reject this paper because (1) the first contribution of the paper is not new as it has already been recognized by a few paper that SGD exhibits two different regimes; (2) this paper makes the debate of large-batch training even muddier.
Main argument:
The paper does not do a great job in clarify the debate. Particularly, the authors mixed their observations up with the results of published works, making it hard to identify the contributions of this paper. For example, the two regimes mentioned in the paper has been identified by a few other works and the contribution of this paper is just to verify them again. Also, I find the experiments done in section 3 and 4 are similar to previous works and even the conclusions are similar. The only new observation I'm aware of in these two sections is that the training loss and test accuracy are independent of batch size in the noise dominated regime.
Back to introduction section, the goal of this paper (as claimed in the beginning of second paragraph) is to clarify the debate. But does this paper really achieves this goal? In terms of learning rate scaling, this paper gets similar conclusions as Shallue et al. (2018). In terms of the difference between vanilla SGD and SGD with momentum, Zhang et al. (2019) already argued that the difference depends on specific batch sizes and SGD with momentum only outperforms SGD in the curvature dominated regime. 
I think the authors should instead focus on the discussion of generalization performance and the observation that training loss and test accuracy are independent of batch size in noise dominated regime. To my knowledge, this part is novel and interesting. 
In summary, I'm inclined to reject this paper given the current version. However, I think the paper is still worth reading if the authors can reorganize the paper and I might increase my score if my concerns get resolved.