How are you testing the accuracy after loading?
If you are using the validation set, did you make sure to call model.eval() before?
Are you also restoring the optimizer for the training after reloading?