AlbyTree:

and now it works even with the interrupted models.


If I understand it correctly, you are not seeing the out of memory issue, if you are not loading the state_dict?
If so, could you push the model to the CPU before saving the state_dicts or use torch.load(checkpoint_path, map_location='cpu)`?