Did you save the optimizer state with the model?
Different optimizers tend to find different solutions so changing optimizers or resetting their state can perturbe training. That is why it can be important to not only save the model parameters but also the optimizer state.
Of course, in this case, that might have nothing to do with the fluctuation of the error that you are seeing.