Solved by tom in post #6 
              
              
                I would try to replace the PIL augmentation by one on GPU using torchvision or eg kornia. 
Also, you would probably do yourself a favour by using Dataset and Dataloader from PyTorch.