For distributed parallel training, we do allocate one process for one GPU to train model.
However, in the dataloader, because it also adopts the multi-processing, the increasing num_workers will increase the GPU memory footprint.
In my opinion, the increased GPU memory footprint due to the increasing num_workers can not help to training (faster of better performance).