I noticed that Pytorch is writing and deleting a lot of data to my main drive and I couldn’t find a way, how to change the drive, that the data is written to. Is that even possible?
In my case for each epoch it generates and deletes about 8 GB of data, which can really hurt my SSD’s lifespan when things run for hours and there are multiply multi-hundred epoch trainings daily. I’d like to save that data either in a RAM disk (if there are speed benefits) or to an HDD.
By the way: I can’t make out, why there is even 8 GB of new data generated per epoch, when the dataset is just about 20 MB big in form of about 2500ish images. I do random flips, rotations and crops, but that can’t cause the temporary data to be bigger by a factor of 400, can it?