Using batches during training comes about as a tradeoff b/w two competing objectives.
Firstly, in terms of the learning algorithm, we would ideally want to backpropagate on every training example. This would allow the optimizer to slightly tweak the NN to best fit every training example. However, this methodology has the obvious drawback of being computational inefficient.
On the other hand, we can compute the loss on the entire dataset and backpropagate only once every epoch. This approach would be more computationally efficient, utilizing the benefits of parallelization. But, this would accumulate the loss for the entire dataset and backpropagate only once every epoch. By using a single batch we would only be able to update once for a pass through the entire dataset and this can limit the network’s ability to learn (especially the edge cases).
Thus, using batches provides the best compromise in terms of computational efficiency and the number of updates required per epoch for the network to best approximate the training data. I would further encourage you to read up on batch gradient descent v/s mini-batch gradient descent.
Also by having multiple batches the optimizer can update the NN multiple times every epoch. This can reduce the overall training time by reducing the total number of epochs required for reaching convergence.
Another thing to consider would be the added memory overhead that is associated with large batch sizes. This can be a massive hindrance, particularly while training on GPUs with limited VRAM.
Hope this helps.