For neural networks, is mini-batching done purely because of memory constraints? I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing power to use entire dataset every time, so mini-batching is the intermediate solution?
If so, does that mean that it's best to use the biggest batch size your machine can handle when training neural networks?
Any insight, references or info would be appreciated!

It is difficult to prove anything and many results assume some network architectures, unless you can use fit everything into 1 batch.
But from my experience, for small batch sizes relative to the dataset, you want smaller learning rate than when using larger batch sizes (intuitively, you want the general direction to be the sum of many small vectors). Problem is nobody knows what is small enough or large enough is, but we have general idea for some network architectures.
Furthermore, with many training algorithms, the learning rates can adapt and it can get really complex to prove anything.
To improve your training speed, do make batch size as large as possible unless you can fit your entire data into one batch, then breaking it down to mini batches may help you break out of the first local minimum you encounter. There is nothing, however, that guarantees that the other minimums are better than your first local minimum. It is an empirical result really...
