I think in your example the creation of prob when rank != 0 should be prob=torch.zeros(1, device=rank).
Your codes:
def worker(gpu, *args):
    ...
    if gpu == 0:
        prob = torch.rand(1).cuda() #.to(device=gpu)
        print(gpu, prob)
    else:
        prob = torch.zeros(1).cuda() # .to(device=gpu)
        print(gpu, prob)

# output
0 tensor([0.1045], device='cuda:0')
1 tensor([0.], device='cuda:0') <- same on cuda:0

Specify device
def worker(gpu, *args):
    ...
    if gpu == 0:
        prob = torch.rand(1, device=gpu)
        print(gpu, prob)
    else:
        prob = torch.zeros(1, device=gpu)
        print(gpu, prob)

# output
0 tensor([0.8787], device='cuda:0')
1 tensor([0.], device='cuda:1')