Facebook Trains ImageNet in 1 Hour
I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time.  You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?It seems to my naive view like it should be "nice" from an accuracy perspective to look at more samples before making an adjustment to the network weights ...?In general, does changing the batch_size hyperparameter make a lot of difference on different problems ...?  Does the right value for
batch size tend to be problem specific and/or network architecture specific?