Thanks for the update.
I cannot find any obvious error, but I would recommend to try to narrow down the error by storing just one state_dict after e.g. 10 epochs, restore it, and check the result.
If you are able to save and load a single state_dict giving you approx. the same model performance, the error might be in your current storing mechanism.
Otherwise we would need to look into the model or training routine.