Carsten_Ditzel:

Have you experienced problems where data is corrupted/mixed up if read with multiple workers from hdf5


No, I didn’t (or I don’t know I did).



 Carsten_Ditzel:

I have 8 CPU cores and a Ti1080 with 11 GB, may I ask how many workers you use in the dataloading process?


To fully utilise 1060GTX I needed only 4 workers. However, I also use V100 card and 10 workers on that machine and I wan’t able to fully utilise this GPU neither using PyTorch nor Keras, so you may be right that there is need for some roadmap from more experienced people how to properly parallelise preprocessing and feed data into GPU. Do you have some useful resources?