You could profile the data loading using the ImageNet code 3 and check if that’s really the bottleneck.
If so, you might take a look at this post 4, which explains potential bottlenecks in the data loading pipeline.