Facebook Trains ImageNet in 1 Hour
I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time.  You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?<p>It seems to my naive view like it should be "nice" from an accuracy perspective to look at more samples before making an adjustment to the network weights ...?<p>In general, does changing the batch_size hyperparameter make a lot of difference on different problems ...?  Does the right value for
batch size tend to be problem specific and/or network architecture specific?