The code snippet looks fine to me.
Could you provide some information regarding the training? I assume you are observing the training loss and accuracy, save the model, and after resuming you see a higher price training loss.
I created a small example 28 some time ago and could not reproduce the issue (from another thread).
Probably we would have to inspect other parts of your code to see if something goes wrong.