You can refer to the example code in this reply.

  
    
    
    Save checkpoints trained on multi GPUs for load on single GPU distributed
  
  
    Hi @Janine, 
This is not related to PyTorch version but the DataParallel (also DistributedDataParallel) class wrapper of PyTorch nn class models. 
DataParallel encloses the original model as it member variable, self.module. 
In case you need both single-GPU and multi-GPU model training, you can change saving/loading behavior with if statements. 
For example, 
if isinstance(G, nn.DataParallel):
    torch.save(G.module.state_dict(), model_save_name)
else:
    torch.save(G.state_dict(), model_save_…