Gotcha.  If testing a larger batch size isn’t practical, my next step would be a test with torch.utils.bottleneck.  NOTE: you only need to run through a handful of training and validation passes during your bottleneck testing.  I made the mistake of trying to run it through a full training run the first time I tried using it.  It took forever and used up all 64GB of my RAM before finishing… 
–SEH