I’m not familiar with Sun Grid Engine, but if multiple nodes in the system can talk to each other over TCP, you can follow this tutorial: https://pytorch.org/tutorials/intermediate/dist_tuto.html 17. You probably want to use the TCP initialization method as described here: https://pytorch.org/tutorials/intermediate/dist_tuto.html#initialization-methods 6.