cc @Kiuk_Chung
As of today DDP does not officially support running jobs with different number of GPUs on different machines. One workaround might be to use the CUDA_VISIBLE_DEVICES environment variable to split 8 GPUs into two sets and then launch two DDP processes.