Solved by Yanli_Zhao in post #2 
              
              
                just do like this: 

Define Model
model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model) 
model.cuda(args.gpu) 
model_DDP = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True) 

save and load Model
torch.save(model_DDP, tmp.name) 
model_DDP = to…