Hm, haven’t seen this before. There’s a related thread here:

  
    
    
    NCCL/CUDA error running on multiple GPUs: torch.cuda.nccl.NcclError: System Error (2) 
  
  
    Hi, 
I’m trying to run several GAN architectures in PyTorch (running pytorch.0.2) in a docker container (started using nvidia-docker) on 4 NVIDIA K80 GPUs. The code runs fine in CPU or single GPU mode. However when trying to use multiple GPUs it crashes. I’ve tried this experiment for both the DCGAN and the DiscoGAN architectures. 
The error I get in each case (torch.cuda.nccl.NcclError: System Error (2)) is below. Any ideas? 
Traceback (most recent call last): 
File “main.py”, line 41, in  
mai…
  


But maybe there the issue is due to some other problem. When i save models, I usually convert them to CPU first, because that keeps them most compatible. Then, when I reload them, I can put them on any GPU that are currently available.
Maybe you didn’t do that when you saved the model and when you try to load it, the original (or maybe not all) of the cuda devices (or the main cuda device) are not available?