Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best fixed batch-size. At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function. Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success. After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size. Experimental results show that the RMGD achieves performance better than the best performing single batch size. It is surprising that the RMGD achieves better performance than grid search. Furthermore, it attains this performance in a shorter amount of time than grid search.
The authors consider the problem of determining the minibatch size for SGD by first fixing a set of candidate sizes, and then learning a distribution over those sizes using a MAB algorithm. A minibatch size is first sampled from the distribution, then one training epoch is performed. A validation error is then computed, and if it is lower than that of the last epoch, the cost of the minibatch is taken to be zero (otherwise one), and the distribution is updated. This is Algorithm 1.
In Section 4.2, they prove a regret bound, but I don’t think that regret is really the correct notion, here (although it’s very close). This is a subtle point, so I’ll set up some notation. Let w(b_1, ..., b_t) be the result at the tth epoch, if the batch sizes b_1, …, b_t were used at the 1st through tth epochs. Let y(w,b) be 0 if training one epoch starting at w with batch size b would improve the validation error, and 1 otherwise.
They show (unnumbered inequality on the middle of page 5) that \sum_t y(w(b_1,...,b_{t-1}),b_t) is close to \sum_t y(w(b_1,...,b_{t-1}),b^*), where b_t is the batch size that was chosen at time t, and b^* is the best fixed batch size. The key point here is that the comparator (the second sum) starts each epoch at the result that was found by their adaptive algorithm, *not* what would have been found if a batch size of b^* had been used from the beginning.
In other words, their result does *not* show that their algorithm is close to outperforming a fixed choice of batch size (for that to hold, the comparator would need to be \sum_t y(w(b^*,...,b^*),b^*)). What they show is similar, but subtly different. They don’t put too much weight on this theoretical result, and in fact don’t even explicitly claim that the comparator in this result is that for a fixed choice of batch size, so really this is a minor issue, but I think that this is something that should be clarified, since it would be easy for a reader to draw an incorrect conclusion.
With that said, their approach is well-motivated, and their experiments seem to show consistent small improvements in performance. I don’t think the performance improvements are totally conclusive, but one of the most appealing properties of their proposal is that it shouldn’t be much more computationally expensive than using a fixed minibatch size. Furthermore, their approach is potentially more robust, since you can presumably be less careful about choosing the set of candidate minibatch sizes, than you would be for choosing only one. So while the experiments don’t show a big improvement, their proposal has other benefits.