I am trying to run pytorch using 100G dataset with 1M image size,
I wrote a mapstyle dataset to get the image from storage, I have a powerful system (96 core)
so I launched a dataset with 90 workers and 1024 batch size,
but my dataloader is still slow and when look at the i/o traffic does not go beyond 250M/sec
Seems like at the moment each worker only get one index and load one data point,
I tried passing sampler to fetch multiple data points but did not help the performance  stays the same.
Can some one help me to make the worker fetch more files quickly and improve the performance of the dataloader?