Hello,
I want to do some heavy data augmentations during training, which results into significant slow down of the training as even having 20-24 workers does not help in preparing a batch of size 32 so GPU would not stale. I am thinking about having an architecture where the data is prepared continuously on several worker nodes and then fed to the machine with GPUs for training. I am curious if there is any existing frameworks which allow doing something like this?