My model training is bottlenecked by IO, and I stream data from S3 using AWS wrangler. I only use 1 GPU for my model training. My machine has 8 GPUs, and I found when I run multiple training jobs, e.g. 4 jobs on my machine, the total IO of the machine increased 4 times. This demonstrated that the S3 throughput and network do not bottleneck my IO.
It seems that a single data loader has a bottleneck, and I am curious about where the bottleneck comes from. Is there a way to run 4 data loaders to load different part of the dataset simultaneously in one training job and interleave them together, to increase the IO performance of the data loading?