I’m trying to understand what the differences are in using DataParallel vs increasing the num_workers in the DataLoader.
It seems that DataParallel divides the batch uniformly across the available GPUs, allowing the forward and backward passes to be done on each split up batch in parallel.
But what does increasing the num_workers in DataLoader do? That is, does each process generated to consume a new batch?