Hi everyone,
I hope to do data-augmentation ‘on-the-fly’. According to this link: Fast data loader for Imagenet,  data-augmentation can significantly slow down the training process.
I am curious is there a way to use one process to augment data and save augmented  ‘dataLoader’ in separate files, use another process to load the saved ‘dataloaders’ and train the network ? The two processes can run simultaneously so that the overload of data augmentation can be alleviated.
Another question, is there a way to save ‘dataLoader’ as a file (e.g., LMDB) to the hard-disk? I need to process a large database which cannot fit into the RAM.
Thanks,
Yuhang