Hi Alex!



 Pkpklong:

The larger the batch size, the more the GPU can be utilized, but in exchange my model converges to a local minima due to a lack of noise in the batch.
So my question is, is there some way where I can achieve the same performance gains of using a larger batch sizes while retaining the convergence  results of smaller batch sizes?


You could try just adding some noise to the gradients computed
using the larger batch sizes.
(For additive noise this is the same as subtracting learning-rate
times noise from your parameters after the SGD optimizer step.)
You could consider applying multiplicative noise to you gradients
so that the size of your noise scales, in effect, with the size of
your gradients.
You would now have three interacting parameters to play with:
batch size, size (and type) of noise, and learning rate.
(I don’t know how well this is likely to work, but it is certainly
something people experiment with.)
Good luck.
K. Frank