Do I understand the issue correctly that your model is performing as expected if you are training and restoring the model on the GPU, but fails if you are training on the GPU and restoring on the CPU?
If so, which PyTorch version are you using? And if not the latest one, could you update it to the latest stable or nightly release?
A while ago (maybe ~6 months) I’ve debugged an issue where the compression during torch.save created invalid data when e.g. nn.Embedding layers were stored from the GPU (it’s already fixed). If you could resave the model, could you disable the compression via torch.save(..., _use_new_zipfile_serialization=False) and check if reloading this state_dict would work?