For processes: A,B
Each has to do
(1) distributed communication (e.g A–>C , B->D)
(2) share parameters (A<->B)
For the distributed communication I need different ranks (Im currently using cuda-aware MPI).
Even if there is an option to use distributed communication with threads (i think that there is in mpi, not sure if Pytorch supports it), in python it is a pretty bad Idea.