Published as a conference paper at ICLR 2018 DON'T DECAY THE LEARNING RATE, INCREASE THE BATCH SIZE
It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate and scaling the batch size B ∝ . Finally, one can increase the momentum coefficient m and scale B ∝ 1/(1 − m), although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes.
## Review Summary
Overall, the paper's paper core claim, that increasing batch sizes at a linear
rate during training is as effective as decaying learning rates, is
interesting but doesn't seem to be too surprising given other recent work in
this space. The most useful part of the paper is the empirical evidence to
backup this claim, which I can't easily find in previous literature. I wish
the paper had explored a wider variety of dataset tasks and models to better
show how well this claim generalizes, better situated the practical benefits
of the approach (how much wallclock time is actually saved? how well can it be
integrated into a distributed workflow?), and included some comparisons with
other recent recommended ways to increase batch size over time.
## Pros / Strengths
+ effort to assess momentum / Adam / other modern methods
+ effort to compare to previous experimental setups
## Cons / Limitations
- lack of wallclock measurements in experiments
- only ~2 models / datasets examined, so difficult to assess generalization
- lack of discussion about distributed/asynchronous SGD
## Significance
Many recent previous efforts have looked at the importance of batch sizes
during training, so topic is relevant to the community. Smith and Le (2017)
present a differential equation model for the scale of gradients in SGD,
finding a linear scaling rule proportional to eps N/B, where eps = learning
rate, N = training set size, and B = batch size. Goyal et al (2017) show how
to train deep models on ImageNet effectively with large (but fixed) batch
sizes by using a linear scaling rule.
A few recent works have directly tested increasing batch sizes during
training. De et al (AISTATS 2017) have a method for gradually increasing batch
sizes, as do Friedlander and Schmidt (2012). Thus, it is already reasonable to
practitioners that the proposed linear scaling of batch sizes during training
would be effective.
While increasing batch size at the proposed linear scale is simple and seems
to be effective, a careful reader will be curious how much more could be
gained from the backtracking line search method proposed in De et al.
## Quality
Overall, only single training runs from a random initialization are used. It
would be better to take the best of many runs or to somehow show error bars,
to avoid the reader wondering whether gains are due to changes in algorithm or
to poor exploration due to bad initialization. This happens a lot in Sec. 5.2.
Some of the experimental setting seem a bit haphazard and not very systematic.
In Sec. 5.2, only two learning rate scales are tested (0.1 and 0.5). Why not
examine a more thorough range of values?
Why not report actual wallclock times? Of course having reduced number of
parameter updates is useful, but it's difficult to tell how big of a win this
could be.
What about distributed SGD or asyncronous SGD (hogwild)? Small batch sizes
sometimes make it easier for many machines to be working simultaneously. If we
scale up to batch sizes of ~ N/10, we can only get 10x speedups in
parallelization (in terms of number of parameter updates). I think there is
some subtle but important discussion needed on how this framework fits into
modern distributed systems for SGD.
## Clarity
Overall the paper reads reasonably well.
Offering a related work "feature matrix" that helps readers keep track of how
previous efforts scale learning rates or minibatch sizes for specific
experiments could be valueable. Right now, lots of this information is just
provided in text, so it's not easy to make head-to-head comparisons.
Several figure captions should be updated to clarify which model and dataset
are studied. For example, when skimming Fig. 3's caption there is no such
information.
## Paper Summary
The paper examines the influence of batch size on the behavior of stochastic
gradient descent to minimize cost functions. The central thesis is that
instead of the "conventional wisdom" to fix the batch size during training and
decay the learning rate, it is equally effective (in terms of training/test
error reached) to gradually increase batch size during training while fixing
the learning rate. These two strategies are thus "equivalent". Furthermore,
using larger batches means fewer parameter updates per epoch, so training is
potentially much faster.
Section 2 motivates the suggested linear scaling using previous SGD analysis
from Smith and Le (2017). Section 3 makes connections to previous work on
finding optimal batch sizes to close the generaization gap. Section 4 extends
analysis to include SGD methods with momentum.
In Section 5.1, experiments training a 16-4 ResNet on CIFAR-10 compare three
possible SGD schedules: * increasing batch size * decaying learning rate *
hybrid (increasing batch size and decaying learning rate) Fig. 2, 3 and 4 show
that across a range of SGD variants (+/- momentum, etc) these three schedules
have similar error vs. epoch curves. This is the core claimed contribution:
empirical evidence that these strategies are "equivalent".
In Section 5.3, experiments look at Inception-ResNet-V2 on ImageNet, showing
the proposed approach can reach comparable accuracies to previous work at even
fewer parameter updates (2500 here, vs. ∼14000 for Goyal et al 2007)