We don’t provide a way of doing this natively in PyTorch. Writing a simple bash script to do this should be doable though (take a list of hostnames, ssh into each one, copy over pytorch program, and run). You could also use slurm/mpirun if those are available in your environment.
If you decide to use torchelastic for distributed training, then there are some plugins available to simplify training in the cloud: elastic/kubernetes at master · pytorch/elastic · GitHub