The issue sounds quite strange, since you should see a NaN output if any parameter of the model is already invalid (Inf or NaN).
Could you check the state_dict after saving it by directly re-loading it and comparing the values for each parameter? Also, which PyTorch version are you using? We’ve had an issue in PyTorch ~1.6 or so, which created invalid values while saving a state_dict containing parameters on the GPU.
If you are using an older PyTorch release, try to use _use_new_zipfile_serialization=False in torch.save or update to the current stable release.