Could you try to profile the data loading pipeline, e.g. using this code 23, or alternatively remove the data loading and feed a random input to your model?
This would give you some information, if the data loading is the bottleneck, which might be the case given the low utilization of the device.