Hello,
I’d like your opinion on some approaches for applying preprocessing to images for deep learning (eg: semantic segmentation).

Important note :

Take for account that this would be performed in a pipeline therefore every new training would re-apply this (consider the disk to be emptied between training for instance). Also take for granted that we have access to cpus and gpus.


Approach 1 : Batch Transforms, Real time data augmentation
This approach would essentially apply the transforms like resize, rescale (/255) and toTensor in a batch and save the transformed images on disk for training.
Then during training, only the data augmentation would be left to be applied (for the training set only).
TLDR: Transforms (images) => onDisk => Training + Data aug on the fly

Approach 2 : Batch Transforms, Batch data augmentation
This approach would apply the transforms like resize, rescale (/255),toTensor and the data augmentation for the training set then save it on disk for training.
Then during training, we’d use a dataset with transforms=None meaning no Transforms are applied, it would simply read the already processed images.
TLDR : Transforms(images) => Data Aug(images) => onDisk => Training

Approach 3 : Batch Data augmentation, Real time Training + Transforms
Same principle but for data aug (for training set)
Data augmentation (images) => onDisk => Training + Transforms on the fly

Approach 4 : Real time transforms, real time data augmentation
Here nothing is saved to disk besides the raw train/valid/test images.
Training + Transforms + Data augmentation on the fly.
Important things to keep in mind :

Keep in mind that we could decide to perform preprocessing on the cpu or gpu if we use any batch approach, this might or might not be advantageous.
We also might want to consider if having them on disk is useful to save it for reproducibility and lineage (in case we want to go back from a model in production for instance and see what it was trained on).
We also must keep in mind that if we perform the data aug on the fly then it is re-applied after each epoch therefore this might or might not improve model accuracy.

So what do you guys think ?