If I understand the issue correctly, you are getting worse training and test results by loading the trained model.
Could you pass a constant input to the model (e.g. torch.ones) after training it (and before saving the state_dict) and save the output as a reference, and compare it to the result in your test script after loading the state_dict?
If these results are equal, then I guess the difference comes from the data loading and processing in both scripts.