Maybe your bottleneck is I/O bound.
Could you try to increase the number of workers and see, if your training speeds up?
You might also time your data loading using this example from the ImageNet training 39.