Ilia_Karmanov:

However, I am using a cluster-management system and the admin would prefer I submit only command and hence it would have to be the same command.


I’m assuming you mean you’d like to use the same command on all the nodes to spawn the processes. You can use this command on all nodes, but we need to do something to handle the rank:
python -m torch.distributed.launch --nproc_per_node=1 --nnodes=3 --node_rank=<rank> --master_addr=10.47.164.34 --master_port=12345

You probably need some way of passing the appropriate unique rank in. Does your cluster-management system allow for maybe using environment variables which are different per nodes? If so, you can pass in the node_rank via an environment variable.
Another option might be using TorchElastic (which is a fault tolerant wrapper around DDP): https://pytorch.org/elastic/0.2.1/quickstart.html 4. TorchElastic figures out the rank automatically, so you can use the same command on all nodes.