For neural networks, is mini-batching done purely because of memory constraints? I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing power to use entire dataset every time, so mini-batching is the intermediate solution?
If so, does that mean that it's best to use the biggest batch size your machine can handle when training neural networks?
Any insight, references or info would be appreciated!

It's complicated.  We're trying to balance multiple effects.  Yes, you are right in your reasoning as far as it goes, but there are other considerations as well.
GPU utilization favors larger batches, up to a point.  Larger batches allow us to parallelize the computation -- up until you max out the capacity of your GPU.  GPUs are designed to do many copies of the same computation in parallel.  So, a mini-batch of only one sample is usually wasteful: it doesn't use the full capacity of your GPU.  Depending on your GPU and the network, there will be some number of copies of the neural network it can run in parallel.  It can be beneficial to set a batch size around that number.  For instance, suppose your GPU can support a batch size of up to 64.  Then using a batch size of 32 will only use 50% of the capacity of your GPU, wasting its power and causing training to take 2x longer than necessary; using a batch size of 64 will use its full capability and make training time the fastest possible; and using a batch size larger than 64 won't offer any further speedups.
Mathematical optimization favors larger batches, in some sense.  One way to view training a neural network is as solving a mathematical optimization problem: namely, we're trying to find network weights that minimize the training loss.  Gradient descent gives you some kind of approximation to the optimal solution.  From that perspective, yes, if you fix the number of iterations of gradient descent, then the larger the batch size, the better the quality of the solution gradient descent finds (the lower the training loss).  So, yes, from this perspective, what you write is absolutely correct.
Generalization favors smaller batches.  It turns out there is another, subtler issue: generalization.  It turns out that if you try as hard as you possibly can to find the optimal solution to the learning optimization problem (and in particular, run gradient descent for as long as you can stand to), neural networks tend to overfit to the training data and thus generalize poorly to other data.  This is bad.  There are various methods for combating overfitting, but one of the most fundamental methods used in neural networks is "early stopping": namely, we stop gradient descent early, before it has reached an optimal solution to the optimization problem.  This is bit counter-intuitive, because it means that we are deliberately accepting a poorer solution to the optimization problem.  It is surprising that it is beneficial to do so, but it turns out that early stopping acts as a form of regularization and helps the neural network generalize better.  As far as I know it is an open question exactly why early stopping helps, but it does.
And, as a result, using very large batches is in tension with early stopping.  My understanding is that using very large batches has a somewhat similar effect to running gradient descent for a long time, and thus may cause overfitting.  Intuitively, running for 1000 iterations of gradient descent with a batch size of 128 is vaguely comparable to 2000 iterations with a batch size of 64, so you can perhaps see why, for a fixed number of iterations of gradient descent, larger batch sizes pose a greater risk of overfitting.
Just to add to the complications, in practice we don't hold the number of iterations fixed.  Instead, we hold fixed the batch size * the number of iterations.  This is typically measured in terms of the number of epochs, which is measured as the batch size * the number of iterations / the size of the training set.  For a fixed number of epochs, it turns out that using a smaller batch size gives a slightly better solution to the optimization problem, so we have some incentives to use smaller batch sizes.  This effect might be relatively minor, for the ranges of batch sizes used in practice, but if you tried to put the entire training set in one batch, this might potentially lead to a degradation in the effectiveness of the learned model.  (The number of iterations determines how long it takes to train the model on a single GPU, in wall-clock time; the number of epochs determines how much energy or computation it takes to train the model, in total.)
Summary.  We are dealing with competing concerns.  From the perspective of generalization and overfitting, it might be best to use a batch size of 1, or small batches.  But this would make very inefficient use of GPUs, and would make training take prohibitively long.  From the perspective of using your GPU fully and making training complete in a reasonable amount of time, we are incentivized to use a batch size large enough to fully utilize all of the GPU's parallel computing units.  So, in practice, it's common to choose batch sizes somewhere around there.
