After a couple of weeks of intensively working with pytorch, I am still wondering what the most efficient way of loading data on the fly is, i.e. not considering loading the entire data into RAM.
The dataloader tutorial 101 reads in csv files and then pngs in every call to getitem().
I used to use hdf5 but cannot get rid of some nasty bottlenecks plus the looming danger of receiving corrupted data due to hdf5’s multiprocessing issues, although I included several checks to exclude those errors.
pytorch offers a “fakedata” dataset that exhausts my TitanX during training for about 65% but without ever dropping down which is quite nice. Apparently it accesses pil images, but does that mean, that images are really saves as pil on disk?
Then there is the option of saving the data via torch.save() prior to the training process. Would that be beneficial under certain circumstances?
My intention is to get rid of the regular GPU exhaustion drops after every epoch and I have tried every alley so far, but without much success.
I even did load my entire set into RAM, roughly 2 GB but I still got the drops…