To see if the problem comes from the model loading, can you try to look if the weights are the same before being saved and after loading a model.
For example looking at your last layer :
agent.dqn.layers[-2].weight
See if the matrixes are the same.
If yes, then it comes from the way you are evaluating your model and we will need to see what you do there.