Different batch size = different average gradients per batch. Depending on the total size of the data and the training time, this can have a big effect on final performance. Gradient accumulation should help with that.