I am trying to adapt some existing training code to run on multiple GPUs on the same node using DistributedDataParallel.  I have a main_worker_function which sets up the dataset, model, etc then runs training and validation. I run either a single version of this to run on a single GPU, or call it using mp.spawn to run with DDP:
if is_distributed:
    print("Using multiprocessing...")
    devids = ["cuda:{0}".format(x) for x in config.gpu_list]
    print("Using DistributedDataParallel on these devices: {}".format(devids))
    mp.spawn(main_worker_function, nprocs=ngpus, args=(ngpus, is_distributed, config))
else:
    print("Only one gpu found, not using multiprocessing...")
    main_worker_function(0, ngpus, is_distributed, config)

Currently it seems like I obtain no increase in running speed when running on two GPUS with DDP, and upon closer inspection it seems to be because fetching a new validation iterator takes an enormous amount of time. To check this I replaced my training function with:
time0 = time()
print("Fetching iterator")
val_iter = iter(self.data_loaders["validation"])
time1 = time()
print("Done in time: ", time1 - time0)

val_iter = iter(self.data_loaders["validation"])
time2 = time()
print("Done in time: ", time2 - time1)

Running the above code on a single GPU and with num_workers = 4 yields outputs:
Fetching iterator
Done in time:  0.40811872482299805
Done in time:  0.7533595561981201

Running the same code on a single GPU and num_workers=4, but spawned as single subprocess with  mp.spawn yields:
Fetching iterator
Done in time:  49.32914686203003
Done in time:  50.69797325134277

As far as I can tell it has something to do with the spawning of the worker functions, because spawning a subprocess with num_workers=2 yields:
Fetching iterator
Done in time:  25.56034803390503
Done in time:  28.712769746780396

But I’m not sure what behaviour is changing between single and multiprocessing mode to cause this increase in time, and I’m not quite sure how to check (I’ve been consulting https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading 1 but I’m not sure if it specifies any difference between default and multiprocessing worker spawning). Note that the initialization of the dataset and sampler should currently be performed the same (I was previously using a DistributedSampler in DDP mode, but this problem persists regardless of or not this sampler is used).