If you guys training with more than 1 GPU, this might occured when your resume operation all do on the master GPU.
This can be happen such as statedict stored all from master GPU, your optimizer all store from master GPU etc.
So that all will fail on your multi GPU training processes.