I see, then you will have to write your own launch script since torch.distributed.launch assumes you have the same configuration per node (i.e. launching the same number of processes per node, having same number of gpus per node).