For neural networks, is mini-batching done purely because of memory constraints? I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing power to use entire dataset every time, so mini-batching is the intermediate solution?
If so, does that mean that it's best to use the biggest batch size your machine can handle when training neural networks?
Any insight, references or info would be appreciated!

so you have to understand that neural networks are used as 'computation-bound' statistical models. ie they are solving the problem how accurate can I get with a budget of X compute-hours.
reasoning with a mental model of modelling in under an hour will lead you astray.
minibatch is a tradeoff of stochasticity and iteration speed vs accuracy.
you will be able to achieve more iterations in the same time if you have a smaller minibatch - iteration speed
(batch) gradient descent will get stuck in local minima. so some stochasticity is beneficial.
stochastic gradient descent (ie updating after each sample) will be too noisy (ie the single sample gradient is too far from the batch gradient)
somewhere in between is therefore beneficial. the law of large numbers would suggest that accuracy of the estimate only scales with square root of the number of samples - halving the error requires 4 times the samples, quartering the error requires 16 times the samples (and therefore corresponding computation time...)
