My data loader load a large amount of Numpy arrays(about 150 GB) as input data from disk, each .npy array contains a [32571025] NumPy array.
The batch_size is 8, num_workers is 8(I can’t load a large batch, because it will raise a CUDA out of memory error). The training speed is same as when I use a batch_size of 4. So I check the NVIDIA-smi and found that GPU util is always at a low level. Is there any solution to speed up my training?
I think loading large .npy files slow slow down the process, because when I load a small size of Numpy array(eg.2*16384 size) things would be fine.
Is there any ways to load Numpy array like data faster?
Many thanks!