As a quick check: could you compare the outputs of the model (after calling model.eval()) using a constant input (e.g. torch.ones) before saving and after loading the model?
If these values are equal, the difference might come from the optimizer (it seems you are storing its state_dict, but are you also loading it properly?) or the input data (are you changing any preprocessing etc.).