Solved by yoshitomo-matsubara in post #2 
              
              
                Hi @AndreaSottana 
When using the distributed training mode, one of the processes should be treated as the main process, and you can save the model only for the main process. 
Check one of the torchvision’s examples, which will give you a good idea for your problem. 
Say you’re using two GPUs in the…