Could you check the memory usage of the GPU before running the “resume training” script?
CTRL-Z will suspend, but not kill the process so you should make sure that you have enough memory on your device for further training.
For the state_dict error:
If seems you have stored more that the model.state_dict() in the checkpoint, but are trying to load this checkpoint directly via model.load_state_dict.
Try to index the checkpoint dict via:
model.load_state_dict(checkpoint['model_state_dict'])