I had similara issues:
Loaded model is not the same - PyTorch Forums 1
So what you can check, if you use a scheduler which also contains the information of last_epoch.
Otherwise you can check if you calculate your loss values in model.eval(). This makes a difference especially if you have dropout layers.
If everything is fine, than there might also be other non-deterministic reason. You can read this up here:
Reproducibility — PyTorch 1.7.0 documentation 2