You don’t need to perform the data augmentation on the fly.
You could store to disk all the augmented data, and load them as if it was the full dataset.
The process that you mentioned might be a bit risky, as you might have the situation where you try to load a file that has already been deleted by the OS, but it could be done.
The way we currently do is to use several threads to perform data loading and augmentation, and that significantly speeds-up the process.
It is totally possible to use LMDBs or HDF5 files to store your data, all you need to do is to write a Dataset that loads from those files.
You can find an example of using LMDB here https://github.com/pytorch/vision/blob/master/torchvision/datasets/lsun.py 188