Perhaps I am misunderstanding the question, but the rank is passed into the method and the rank argument is there in the code snippet you sent. For a third party backend you need to register it and perform init process group on all the ranks (Distributed communication package - torch.distributed — PyTorch master documentation). Then you can use the collectives as usual.