Without knowing what exactly your code does, I can think of the following other reason:

If you are using Adam optimizer, your learning rates that you set from your script might mess things up. Because Adam inherently has a square root decay of learning rate. So at the restart of training, even though reloading the optimizer state dict will resume training, I am guessing you again set you annealed learning rate. This might cause your problem.