@mrshenli I have some questions regarding num_workers in dataloader when the model is trained in Distributed scenario:


Does DistributedSampler or Distributed training in general show some specific behavior for multiple worker processes for data-loading? In other words, do I need to reduce the number of data_loading workers when using Distributed training? Does distributed training increase the data_loading workers?


When we do DataParallel with num_workers>1, do the data-loading workers are specified explicitly for specific GPUs or data-loading worker processes runs separately?