Hi everyone,
If all_gather are used more than once,  the next cuda() function will deadlock. And GPU-Utils are 100%.
Details:
I wrapped all_gather in collect:
def collect(x):
    x = x.contiguous()
    out_list = [torch.zeros_like(x, device=x.device, dtype=x.dtype)
        for _ in range(dist.get_world_size())]
dist.all_gather(out_list, x)
return torch.cat(out_list, dim=0)

Next:
a_all = collect(a)
b_all = collect(b)
c = torch.rand(10,10).cuda() # deadlock! No error report.

The deadlock doesn’t happen if  I don’t collect b_all or don’t use cuda().
My code runs on 4 GPUs using DDP.
Pytorch version: 1.4.
This problem has been confusing me for a couple of days. I’ll appreciate any help!