I am not familiar with submitit so I am unsure of how to validate the number of GPUs that is using. Before init_process_group can you also try printing the value of torch.cuda.device_count()? (torch.cuda.device_count — PyTorch master documentation 3). This may help to narrow down why there are any GPUs detected and whether this is an issue in the distributed package.