I guess the problem looks to be in my dataloading. I tried to reduce the random sampling in that by doing it one time and saving in a pickle file. So now I have around 3 million samples in the pickle file. If I want to utilize 8 GPUs properly, I tried using a batch size of 200 and 16 queues, even then the problem persisted. So, I decreased the batch size to 32 for 8 GPUs, now the frequency for all GPU being utilized as 0 goes down. But at the same time, the percentage utilization is 10% because essentially I am giving 4 samples per GPU.
Basically, the multiprocessor is not efficient in keeping the batches ready for processing. Can you help me by sharing a dataloader which is implemented for multiGPU processing?