Sorry for being not clear enough.
I mean setting your model to eval.
So let me understand your workflow.
You train your model and the accuracy is good for the training set.
You evaluate this model, the accuracy is still good, and you save it.
After loading the model you train it again.
How is the training accuracy then?
When evaluating again the accuracy is bad.
Did you use an adaptive optimizer (Adam, etc.)?
If so, did you take care of lowering the learning rate?
Did you also save the optimizer and reloaded it?