Hi @Dawny33,
I recently play with distributed pytorch, and I can give you some pointers here, but I’m not sure if you’ve already figured this out.
I’m not sure if you can create multiple processes on a single machine by running init_process_group on different threads (it works in MPI) but you can try that out and let me know. However, you can definitely run distributed version of this example on a distributed cluster (with more than one EC2 instances).
To do that, the first thing you need to do is to set up your own cluster on EC2, which means you need to let all your worker nodes ssh-able from master node. Here is a good tutorial for doing this (http://mpitutorial.com/tutorials/running-an-mpi-cluster-within-a-lan/ 15). Then, if you look at the example code, it uses tcp to init the cluster, so you will need both to set the init_method to private address of your master node (with a self-defined port e.g. 23456), and to set the rank for each node in your cluster. Like it was described here (http://pytorch.org/docs/master/distributed.html#tcp-initialization 27).
After setting all these things up, run the code on each node (you may want to write a script to simplify this step), then you should get what you want.
Hope this one helps somehow.