One requirement for DistributedDataParallel is that, you need to set device_ids 15 properly or use CUDA_VISIBLE_DEVICES env var to configure them properly to make sure that one process only works on one GPU. Otherwise, by default, each process will try to use all visible GPUs.