My suggestion would be to profile the pipeline. Is GPU utilization near 100% throughout training? If it is not, you could have some sort of preprocessing bottleneck (I/O or CPU bound). Does it make sense in your case?