It’s not really a “new dataset”, just the ordering is different! It might be related to data augmentation, in the sense that the mini-batches received by the network are different each time, even though the images themselves don’t change.
Indeed, the gradients won’t be the same, the convergence won’t be the same, but that’s the magic behind stochastic gradient descent! It’s not realistic to train on the whole dataset due to the deep networks used now and memory constraints, but online training with only one sample at each iteration also isn’t, because the gradients will be too different between successive iterations. SGD is a good tradeoff and has shown its performance for quite some time now