These issues are often caused by a difference in the model usage or the data loading.
To isolate the root cause further, you could store the outputs of the trained model using a static input (e.g. torch.ones) after calling model.eval() and before saving the state_dict().
Afterwards, load the model in your inference script and compare the reference outputs to a new run using the same static inputs. If these outputs differ, the difference would come from the model itself and you could debug further what might be causing the difference (missing keys in the state_dict etc.).
On the other hand, if the outputs are equal (up to floating point precision), you could compare the data loading pipelines in your training and inference scripts and check, if the same preprocessing was applied (e.g. normalization etc.).