Have you checked if dataloading is the bottleneck, as I have seen usually this is the reason for low utilization.

How many workers are you using for dataloading?
How are the images stored ? How much time on average does it take to load the image from disk ?
Are the transforms very time consuming ?