Hey @bing
I see a few errors in the code snippet:

It is if __name__=="__main__" instead of if __name__=="main". I think this is also the reason why you see no output, as nothing is executed.

mp.spawn will automatically provide the rank as the first argument to the target function, so the train function needs to take one additional argument before world_size. Please check the API doc here: https://pytorch.org/docs/stable/multiprocessing.html#torch.multiprocessing.spawn 15

Don’t need to move the loss function to gpu as it does not contain any parameters.gpu = args.world_size
...
criterion = nn.CrossEntropyLoss().cuda(gpu)


You need to use rank instead of the current device, otherwise all processes will use the same device if you didn’t set CUDA_VISIBLE_DEVICE or run torch.cuda.set_device(rank)if use_cuda:
             data, target = data.cuda(), target.cuda()



I tried to fix all above and run your code, but then realized I don’t have the dataset. It would be helpful to have a minimum repro. Or is it hard to create a min repro, I would suggest to expand this example 20 to create your own application.