If I understand the issue correctly, you are seeing a performance drop after loading a saved model and evaluating it on the same dataset?
If so, are you calling model.eval() while the evaluation is performed? If that’s the case, could you pass a static input to the model before saving and after loading (e.g. torch.ones) and compare the outputs?