I know this isn’t precisely what you asked about but to make your training phase faster you can always prepare your data ahead of time. So, for example, instead of running transforms for every iteration, just run them once on your entire dataset offline and then train with the resulting set. Yes, you will lose the randomness that you get with every single iteration, but in my experience it doesn’t improve things by a huge margin if you are smart about how you generate your training set offline.
You can even do a bit better than that and allow for some randomness by running a few fast transforms (resizes, flips etc) during each iteration on your otherwise offline-generated training set. That’s the approach I’m using and it seems to be a good middle ground.