Hi Alan,
We use PyTorch 1.0. The dataset is stored on a high speed local M.2 drive. The task exclusively occupies this server.
We tried a lot of different things but nothing really solve the problem. Our current solution is that we downsample the dataset and load the whole dataset into the memory at the beginning. The multithread data loader is removed from the pipeline and the problem is temporally solved. Once we need higher resolution or larger dataset, I will investigate this issue again. Thanks a lot!