You do have your data on SSD, right?
Is it the loading itself or augmentation? I usually try to move all augmentation to the GPU except gathering statistics which I do beforehand and store in a tensor.
For the loading:

What exact image loading backend are you using?
For PIL, be sure to have the  SIMD version: https://github.com/uploadcare/pillow-simd 2

An alternative backend can be set in TorchVision: https://pytorch.org/docs/master/torchvision/index.html#torchvision.set_image_backend 1

You could also use OpenCV or so instead of TorchVision.

Finally, you ask about the bottleneck of where your training spends its time. This is important and leads to reducing the time per epoch.
You probably are aware, but it is worth reminding ourselves every now and then that other things (most prominently the learning rate) impact the overall runtime to a given target metric by changing the number of epochs needed. (And I say this as someone who  does quite a bit of client work of “benchmark-and-optimize-time-per-epoch”.)
Best regards
Thomas