For neural networks, is mini-batching done purely because of memory constraints? I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing power to use entire dataset every time, so mini-batching is the intermediate solution?
If so, does that mean that it's best to use the biggest batch size your machine can handle when training neural networks?
Any insight, references or info would be appreciated!

There is evidence that supports the proposition that it is best to use the biggest batch size your machine can handle. See e.g. Goyal et al. (2018).
However, that paper (and another) reveal optimization difficulties with extremely large batch sizes. There is concern that large-batch optimization converges to "sharp" minima that generalize less well. Goyal et al. present some strategies to avoid those optimization difficulties (e.g. low-training-rate warmup phase, how to adjust learning rate dependent on batch size, training data shuffling each epoch).
