witl:

You are not correct when you say that “the whole point of batches will be ruined”. That’s not the case at all. There are many other situations/advantages of using batches.


As per my understanding, the point of batching is when you cannot move your whole population (dataset in this case), so you take a population sample (a batch) and estimate parameters based on this iteratively as the expectation of the sample parameter will give you population parameters.
So you are perfectly right, if you can move all the dataset onto GPU will be faster and even better at estimating parameters, that is the best case but not a practical one 
Also, when I say the whole point of batching is ruined maybe I should have reformulated it as, ideally, you only need batches when you cannot fit your dataset onto GPU at one go. If you can, that is the best therefore the whole point of a batch is useless.