Not sure what you loss function is, but maybe you are not taking the average over the minibatch size – I guess your dataset is not evenly divisible by the minibatch size and then you end up with 1 minibatch that is much smaller than others in each epoch.