Stochastic Gradient Descent (SGD) methods using randomly selected batches are widely-used to train neural network (NN) models. Performing design exploration to find the best NN for a particular task often requires extensive training with different models on a large dataset, which is very computationally expensive. The most straightforward method to accelerate this computation is to distribute the batch of SGD over multiple processors. To keep the distributed processors fully utilized requires commensurately growing the batch size; however, large batch training often times leads to degradation in accuracy, poor generalization, and even poor robustness to adversarial attacks. Existing solutions for large batch training either significantly degrade accuracy or require massive hyper-parameter tuning. To address this issue, we propose a novel large batch training method which combines recent results in adversarial training (to regularize against "sharp minima") and second order optimization (to use curvature information to change batch size adaptively during training). We extensively evaluate our method on Cifar-10/100, SVHN, TinyImageNet, and ImageNet datasets, using multiple NNs, including residual networks as well as smaller networks for mobile applications such as SqueezeNext. Our new approach exceeds the performance of the existing solutions in terms of both accuracy and the number of SGD iterations (up to 1% and 5×, respectively). We emphasize that this is achieved without any additional hyper-parameter tuning to tailor our proposed method in any of these experiments.
Based on my understanding, this paper describes a novel approach for addressing the large batch training problem. The authors propose increasing the batch size based on reductions in the largest eigenvalue of the Hessian. This is combined with adversarial training using the fast gradient sign method to reduce the total number of iterations required for training and improve generalization performance. Unfortunately, although the numerical results seem quite promising, the algorithm and its explanation and details are not described clearly in the paper, which makes me lean towards rejection. I describe this more fully below:
1. Description of the Algorithm
The description of the algorithm in Section 3.1 is simply not clear, and lacks clear exposition motivating why the algorithm ought to work. To add to this confusion, there appear to be some inconsistencies between the (brief) description of the method and the description given in the Main Contributions and Limitations section in the Introduction. 
As an example, in Section 3.1, the approach for computing the eigenvalue of the Hessian is not described. Which eigenvalue is computed? How is this done? What is the batch size used in this computation? Is it computed over the full training set? The Limitations section briefly describes this (power iteration to tolerance <= 10^-2), but this should be elaborated on in Section 3.1. In fact, the limitations should not be discussed until a clear description of the algorithm is given.
The introduction makes this even more confusing by claiming the second order information is computed by “backpropagating the Hessian operator”. This seems to imply that the 3rd derivative information is computed for second-order information. Later in the Introduction, the authors claim to use Hessian matvecs to perform the power iteration. I believe that the authors mean that the Hessian-vector product is obtained by differentiating the product g’v (a scalar quantity). 
In addition, it was not described how the learning rate is changed in the algorithm. Later in the experiments, none of the additional hyperparameters in the procedure are given, such as the duration factor, kappa, the hyperparameters in the adversarial training, and more. This all ought to be included for completeness.
2. Questions about Details of the Algorithm
If it is indeed the case that the authors are using power iteration to compute the largest eigenvalue, why not use Lanczos method as it typically works better for symmetric matrices? In addition, if the intention was to compute the largest eigenvalue of the Hessian, one must be wary that the power iteration/Lanczos method computes the eigenvalue with largest magnitude (the absolute value of lambda), which may mean that it’s possible that the algorithm is utilizing negative curvature information rather than positive curvature information (particularly in the earlier epochs), which may contradict their intuition based on flat minima. This needs to be addressed.
Secondly, there is no explanation as to why increasing the batch size would lead to consistent decrease in the eigenvalues of the Hessian. This is certainly not true for all optimization problems. Even if the flat minima/sharp minima hypothesis is assumed, is it possible for the iterates after increasing the batch size to still tend towards sharper minimizers after being in a flat region? This intuition and explanation needs to be expanded on (and argued for) in order for the algorithm to make any conceptual sense.
Lastly, why is the duration factor needed to increase the batch size if the eigenvalue condition fails? if the duration factor is removed, how does the batch size evolve? Is it necessary? How is the duration factor tuned?
3. Inconsequential Theoretical Results
The authors also prove a theorem bounding the expected optimality gap with adaptive batch sizes. On closer look, this is a simple adaptation of the result by Bottou, Curtis, and Nocedal [2] and does not utilize any of the algorithmic mechanisms described in the paper. Hence, the theoretical result is not novel, does not provide any additional insight on the algorithm, and could be applied to any adaptive/changing batch size SG algorithm. In my opinion, this ought to be removed. (Assumption 2 is also mentioned in the main paper, but is only described in the Appendix.)
4. Additional Considerations
The paper is missing much work done by Nocedal’s group on increasing batch sizes (some of which utilize the L-BFGS approximation to the Hessian); see [1, 3].
Other relevant work by Sagun, Bengio, and others on large batch training, flat minima, and the Hessian in deep learning ought to be included as well; see [4-7]. 
Lastly, the algorithm demonstrates some significant improvements on the number of iterations. However, efficiency with respect to epochs is not discussed. It may make sense to plot test loss/error against epochs and batch size against iterations for clarity.
Typos/Grammatical Errors:
- Page 2: Should not state “(We refer to this method as ABS)”, easier to include by including (ABS) after Adaptive Batch Size in the beginning of the bullet point.
- Page 6: Section 4: “information” not “informatino”
- Page 6: Section 4: “the” not “teh”
- Page 7: Section 4.1: “confirms” not “confirming”
- Page 7: Section 4.1: no “a” in “a very consistent performance”
Summary:
Overall, although the paper presents some promising numerical results, it lacks a detailed description and explanation of the algorithm to be worthy of publication. It leaves many aspects of the algorithm open to the reader’s interpretation, and I do not believe I could reproduce the results with the information provided. The manuscript needs significant changes to the detail, structure, and writing before it can be considered for publication.
References:
[1] Bollapragada, Raghu, et al. "A progressive batching L-BFGS method for machine learning." arXiv preprint arXiv:1802.05374(2018).
[2] Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." SIAM Review 60.2 (2018): 223-311.
[3] Byrd, Richard H., et al. "Sample size selection in optimization methods for machine learning." Mathematical programming134.1 (2012): 127-155.
[4] Chaudhari, Pratik, et al. "Entropy-sgd: Biasing gradient descent into wide valleys." arXiv preprint arXiv:1611.01838(2016).
[5] Jastrzębski, Stanisław, et al. "DNN's Sharpest Directions Along the SGD Trajectory." arXiv preprint arXiv:1807.05031(2018).
[6] Sagun, Levent, et al. "Empirical Analysis of the Hessian of Over-Parametrized Neural Networks." arXiv preprint arXiv:1706.04454 (2017).
[7] Zhu, Zhanxing, et al. "The Regularization Effects of Anisotropic Noise in Stochastic Gradient Descent." arXiv preprint arXiv:1803.00195 (2018).