Solved by David_Harvey in post #2 
              
              
                Add this line right after the setup(rank, world_size) 
setup(rank, world_size)
torch.cuda.set_device(rank)
for x in range(1, 10000):
...