For neural networks, is mini-batching done purely because of memory constraints? I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing power to use entire dataset every time, so mini-batching is the intermediate solution?
If so, does that mean that it's best to use the biggest batch size your machine can handle when training neural networks?
Any insight, references or info would be appreciated!

It might be useful to consider the extreme cases of mini-batch sizes: using a single sample vs taking all $N$ samples in one go. These are sometimes also referred to as on-line and batch learning, respectively.
For on-line learning (single sample), we could list the following properties (I am probably ignoring a lot here):

*

*The gradient will depend on the sample we consider. This makes the update process stochastic, which is believed to benefit generalisation.

*The updates have to be applied sequentially, since there is no possibility for parallellisation across samples. We first have to apply the update due to the current sample, before we can move to the next.

*After one pass through the dataset, we will have made $N$ updates.

For batch learning ($N$ samples), on the other hand, we have

*

*The gradient is always going to be the same. This is because we are computing the true gradient for the training data, which is deterministic. Of course, this also means that only the training error is guaranteed to go down.

*The computations can be massively parallellised. We can compute the gradient for each sample at the same time and we only have to accumulate them in the end.

*After one pass through the dataset, we will have made $1$ update.

Conceptually, the on-line learning scenario is arguably more attractive.
After all, we get much more updates out of a single pass through the data and generalisation performance should be better.
The batch learning scenario, on the other hand, looks more promising in practice, because it can benefit from parallellisation, reducing run/waiting times.
However, modern datasets can be so large that it has become impossible to use all data at once, which obviously cripples this argument.
In the end, the size of your mini-batches seems to be a trade-off between how long you want to wait for computations and how good you want your model to generalise. At least for small datasets.
If you are working with huge datasets, you will probably end up with mini-batches that consist of a collection of single samples for a subset of all possible classes. As a result, the gradients will be stochastic enough to provide generalisation benefits. Moreover, you might have to wait weeks instead of days if you don't take a sufficiently large batch size to finish a single pass through the data. Also, some architectures rely on a sufficiently large batch size to work properly (e.g. batchnorm, CLIP, ...)
TL;DR: it's a trade-off that probably depends on the size of the data.

Wait a minute... Who said that more updates is better?
That turns out to depend on how you do the updates.
If the gradient for a single sample, $(\boldsymbol{x}_i, \boldsymbol{y}_i),$ would be $$\nabla_\theta L(g(\boldsymbol{x}_i \mathbin{;} \theta), \boldsymbol{y}_i)$$ (with $\theta$ the set of parameters and $L$ some loss function), the gradient for the entire dataset should equal $$\sum_{i=1}^N \nabla_\theta L(g(\boldsymbol{x}_i \mathbin{;} \theta), \boldsymbol{y}_i).$$
If we use each sample exactly once in the on-line setting the $N$ updates should be roughly equivalent to the single update from the full-batch setting.
Especially if the learning rate is small enough (i.e. the individual updates do not change the weights too much).
The figure below illustrates the learning curves when trained on the sum of errors.
Note that performance does not depend on the number of updates (but it does depend on the number of passes through the data).

However, instead of optimising the sum of errors, it is actually more common to optimise the average error. This means that the update in the online setting remains the same, whereas the update for the full batch is reduced by a factor $N$:
$$\frac{1}{N} \sum_{i=1}^N \nabla_\theta L(g(\boldsymbol{x}_i \mathbin{;} \theta), \boldsymbol{y}_i).$$
In this case, the update in the full batch setting is much smaller (in the same scale as a single update in the online setting, if you want).
This also means that more updates leads to better performance.
The figure below illustrates the learning curves when trained on the average error. Note that the performance depends on the number of updates (and not on the number of passes through the data).

