I am still at a loss as to how to get hdf5 working in a multiprocessing environment. I included several NaN checks in my scripts to exclude suspected corruption of data but I am still experiencing inconsistent behaviour across multiple parallel training runs. To be precise. I am working on a GPU cluster with four Ti1080s inside virtualenvs and all I changed from run to run is the number of workers, ranging from 0 to 4 and 8, up to 16. Since I suspected that data might be corrupted if read from the same files although swmr is enabled, I used distinct hdf5 files for all four parallel runs. To be clear: I only ever read data, I do not write any. Still strange behaviour remains present as can be clearly seen by looking at the images.

12.png703×652 78.2 KB

I read in this 28 stack overflow post that although data is read from different files it still may be corrupted due to the way, the hdf5 lib handles access system-wide. Having said that, I thought that this issue would be long gone since that post is dating back to 2016 but apparently it is still prevailing. Concretely I am wondering about the following

Is this issue  in some form related to multiprocessing, i.e. number of workers I use in my training?
If I run the same script on different GPUs each pulling its own data from SSD, are the dataloaders in python somehow intertwined, i.e. interfering with each other or completely unrelated?
I read in the stack-overflow post above that there apparently used to be differences in the build of h5py, if used in a conda environment as opposed to an python virtualenv
If so, how can I make sure I use the h5py thread-safe build within my virtualenv?
do I have to write the respective hdf5 file with swmr mode as well, if I intend to read them with swmr later on?

I hope someone has the experience/knowledge to help me with my issue