Oh, and about:



 Vibhatha_Abeykoon:

deploying this multi-machine model parallel module?
Will there be modifications to torch.distributed.launch? What is the current method to launch this
as we do for DDP?


That’s complex, my personal experience says that you should try to split and group your process functionalities, for example, grouping them by “role”:
(Image from RFC #41425 2)
                    
            
          


This idea comes from “micro services”. It makes your application logic much more clearer to understand. RFC proposal #41546 also contains an automatic role-based launcher implementation to address this issue.
However, role based design is not compatiable with:
if __name__ == "__main__":
    ...
    tensor.to("worker:0/cuda:1")
    # do some computation
    tensor.to("worker:1/cuda:0")

because you are manually specifying every destination and location,