Referring to this old issue https://github.com/pytorch/pytorch/issues/14528 13, which was closed, I need to do communications (all_reduce/reduce/broadcast) in two or more different groups simultaneously. For example, if a process’ rank belongs to say the group g0, it could participate in that communication only.
Essentially, referring to https://github.com/pytorch/pytorch/issues/14528 13, I would need to apply if conditional –
 if local_rank in g0: 
torch.distributed.all_reduce(t, group=g0) 
elif local_rank in g1: 
torch.distributed.all_reduce(t, group=g1)

– for participating in all_reduce so that simultaneous communications could happen in non-intersecting groups.
However, that results in  NCCL INFO Call to connect returned Connection refused, retrying. Whereas, doing these all_reduce operations sequentially, which essentially would imply that the collective communications in different groups are sequentiallized, works fine.
Is there something that I am missing?