On Batch Adaptive Training for Deep Learning: Lower Loss and Larger Step Size
Mini-batch gradient descent and its variants are commonly used in deep learning. The principle of mini-batch gradient descent is to use noisy gradient calculated on a batch to estimate the real gradient, thus balancing the computation cost per iteration and the uncertainty of noisy gradient. However, its batch size is a fixed hyper-parameter requiring manual setting before training the neural network. Yin et al. (2017) proposed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds. We extend the BA-SGD to momentum algorithm and evaluate both the BA-SGD and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classification. Experiments confirm that batch adaptive methods can achieve a lower loss compared with mini-batch methods after scanning the same epochs of data. Furthermore, our BA-Momentum is more robust against larger step sizes, in that it can dynamically enlarge the batch size to reduce the larger uncertainty brought by larger step sizes. We also identified an interesting phenomenon, batch size boom. The code implementing batch adaptive framework is now open source, applicable to any gradient-based optimization problems.
 The authors propose extending the recently-proposed adaptive batch-size approach of Yin et al. to an update that includes momentum, and perform more comprehensive experiments than in the Yin et al. paper validating their approach. The basic idea makes a great deal of intuitive sense: inaccurate gradient estimates are fine in early iterations, when we*re far from convergence, and accurate estimates are more valuable in later iterations, when we*re close. Finding the optimal trade-off between computational cost and expected decrease seems like the most natural way to accomplish this, and this is precisely what they propose. That said, I*m not totally convinced by the derivation of sections 2 and 3: the Gaussian assumption is fine as a heuristic (and they don*t really claim that it*s anything else), but I don*t feel that the proposed algorithm really rests on a solid theoretical foundation. The extension to the momentum case (section 3) seems to be more-or-less straightforward, but I do have a question about equation 15: am I misreading this, or is it saying that the variance of the momentum update .mathcal{P} is the same as the variance of the most recent minibatch? Shouldn*t it depend on the previous terms which are included in .mathcal{P}? I*m also not convinced by the dependence on the "optimal" objective function value S^* in equation 6. In their algorithm, they take S^* to be zero, which is a good conservative choice for a nonnegative loss, but the fact that this quantity is present in the first place, as a user-specified parameter, makes me nervous, since even for a nonnegative loss, the optimum might be quite far from zero, and on a non-convex problem, the eventual local optimum at which we eventually settle down may be further still. Also, the "Robbins 2007" reference should, I believe, be "Robbins and Monro, 1951". These are all relatively minor issues, however. My main criticism is that the experiments only report results in terms of *training* loss. The use of adaptive batch sizes does indeed appear to result in faster convergence in terms of training loss, but the plots are in log scale (which I do think is the right way to present it), so the difference is smaller in reality than it appears visually. To determine whether this improvement in training performance is a *real* improvement, I think we need to see the performance (in terms of accuracy, not loss) on held-out data. Finally, as the authors mention in the final paragraph of their conclusion, some recent work has indicated that large-batch methods may generalize worse than small-batch methods. They claim that, by using small batches early and large batches late, they may avoid this issue, and I don*t necessarily disagree, but I think an argument could be made in the opposite direction: that since the proposed approach becomes a large-batch method in the later iterations, it may suffer from this problem. I think that this is worth exploring further, and, again, without results on testing data being presented, a reader can*t make any determination about how well the proposed method generalizes, compared to fixed-size minibatches.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On Batch Adaptive Training for Deep Learning: Lower Loss and Larger Step Size
Mini-batch gradient descent and its variants are commonly used in deep learning. The principle of mini-batch gradient descent is to use noisy gradient calculated on a batch to estimate the real gradient, thus balancing the computation cost per iteration and the uncertainty of noisy gradient. However, its batch size is a fixed hyper-parameter requiring manual setting before training the neural network. Yin et al. (2017) proposed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds. We extend the BA-SGD to momentum algorithm and evaluate both the BA-SGD and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classification. Experiments confirm that batch adaptive methods can achieve a lower loss compared with mini-batch methods after scanning the same epochs of data. Furthermore, our BA-Momentum is more robust against larger step sizes, in that it can dynamically enlarge the batch size to reduce the larger uncertainty brought by larger step sizes. We also identified an interesting phenomenon, batch size boom. The code implementing batch adaptive framework is now open source, applicable to any gradient-based optimization problems.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose extending the recently-proposed adaptive batch-size approach of Yin et al. to an update that includes momentum, and perform more comprehensive experiments than in the Yin et al. paper validating their approach. The basic idea makes a great deal of intuitive sense: inaccurate gradient estimates are fine in early iterations, when we*re far from convergence, and accurate estimates are more valuable in later iterations, when we*re close. Finding the optimal trade-off between computational cost and expected decrease seems like the most natural way to accomplish this, and this is precisely what they propose. That said, I*m not totally convinced by the derivation of sections 2 and 3: the Gaussian assumption is fine as a heuristic (and they don*t really claim that it*s anything else), but I don*t feel that the proposed algorithm really rests on a solid theoretical foundation. The extension to the momentum case (section 3) seems to be more-or-less straightforward, but I do have a question about equation 15: am I misreading this, or is it saying that the variance of the momentum update .mathcal{P} is the same as the variance of the most recent minibatch? Shouldn*t it depend on the previous terms which are included in .mathcal{P}? I*m also not convinced by the dependence on the "optimal" objective function value S^* in equation 6. In their algorithm, they take S^* to be zero, which is a good conservative choice for a nonnegative loss, but the fact that this quantity is present in the first place, as a user-specified parameter, makes me nervous, since even for a nonnegative loss, the optimum might be quite far from zero, and on a non-convex problem, the eventual local optimum at which we eventually settle down may be further still. Also, the "Robbins 2007" reference should, I believe, be "Robbins and Monro, 1951". These are all relatively minor issues, however. My main criticism is that the experiments only report results in terms of *training* loss. The use of adaptive batch sizes does indeed appear to result in faster convergence in terms of training loss, but the plots are in log scale (which I do think is the right way to present it), so the difference is smaller in reality than it appears visually. To determine whether this improvement in training performance is a *real* improvement, I think we need to see the performance (in terms of accuracy, not loss) on held-out data. Finally, as the authors mention in the final paragraph of their conclusion, some recent work has indicated that large-batch methods may generalize worse than small-batch methods. They claim that, by using small batches early and large batches late, they may avoid this issue, and I don*t necessarily disagree, but I think an argument could be made in the opposite direction: that since the proposed approach becomes a large-batch method in the later iterations, it may suffer from this problem. I think that this is worth exploring further, and, again, without results on testing data being presented, a reader can*t make any determination about how well the proposed method generalizes, compared to fixed-size minibatches.
assistant
 The authors propose extending the recently-proposed adaptive batch-size approach of Yin et al. to an update that includes momentum, and perform more comprehensive experiments than in the Yin et al. paper validating their approach. The basic idea makes a great deal of intuitive sense: inaccurate gradient estimates are fine in early iterations, when we*re far from convergence, and accurate estimates are more valuable in later iterations, when we*re close. Finding the optimal trade-off between computational cost and expected decrease seems like the most natural way to accomplish this, and this is precisely what they propose. That said, I*m not totally convinced by the derivation of sections 2 and 3: the Gaussian assumption is fine as a heuristic (and they don*t really claim that it*s anything else), but I don*t feel that the proposed algorithm really rests on a solid theoretical foundation. The extension to the momentum case (section 3) seems to be more-or-less straightforward, but I do have a question about equation 15: am I misreading this, or is it saying that the variance of the momentum update .mathcal{P} is the same as the variance of the most recent minibatch? Shouldn*t it depend on the previous terms which are included in .mathcal{P}? I*m also not convinced by the dependence on the "optimal" objective function value S^* in equation 6. In their algorithm, they take S^* to be zero, which is a good conservative choice for a nonnegative loss, but the fact that this quantity is present in the first place, as a user-specified parameter, makes me nervous, since even for a nonnegative loss, the optimum might be quite far from zero, and on a non-convex problem, the eventual local optimum at which we eventually settle down may be further still. Also, the "Robbins 2007" reference should, I believe, be "Robbins and Monro, 1951". These are all relatively minor issues, however. My main criticism is that the experiments only report results in terms of *training* loss. The use of adaptive batch sizes does indeed appear to result in faster convergence in terms of training loss, but the plots are in log scale (which I do think is the right way to present it), so the difference is smaller in reality than it appears visually. To determine whether this improvement in training performance is a *real* improvement, I think we need to see the performance (in terms of accuracy, not loss) on held-out data. Finally, as the authors mention in the final paragraph of their conclusion, some recent work has indicated that large-batch methods may generalize worse than small-batch methods. They claim that, by using small batches early and large batches late, they may avoid this issue, and I don*t necessarily disagree, but I think an argument could be made in the opposite direction: that since the proposed approach becomes a large-batch method in the later iterations, it may suffer from this problem. I think that this is worth exploring further, and, again, without results on testing data being presented, a reader can*t make any determination about how well the proposed method generalizes, compared to fixed-size minibatches.