Hi @agolynski ,
I think I’ve managed to solve the issue. I didn’t know that I have to call torch.distributed.gather(...) in non-master processes. So the fix for the issue was basically changing this code snippet:
if torch.distributed.get_rank() == 0:
    gathered_data = [...]
    torch.distributed.gather(tensor=my_tensor, gather_list=gathered_data, group=gloo_group_handle)

to:
if torch.distributed.get_rank() == 0:
    gathered_data = [...]
    torch.distributed.gather(tensor=my_tensor, gather_list=gathered_data, group=gloo_group_handle)
else:
    torch.distributed.gather(tensor=my_tensor, group=gloo_group_handle)

I thought that non-master processes will somehow magically be pinged by the master process to send their tensors.