I see, the problem is with MPI initialization. Could you try instructions here:
https://pytorch.org/tutorials/intermediate/dist_tuto.html 8 (under MPI Backend).
I.e.
1. Replace the content under `if __name__ == '__main__':` with `init_process(0, 0, run, backend='mpi')`.
2. Run `mpirun -n 4 python myscript.py`.