Did you try to remove the data loading and check the GPU utilization using random tensors created on the GPU as described before? This could narrow down the bottleneck of your code and make it easier to isolate it.
Also, you could profile your code as described in this post 1 using Nsight Systems or the PyTorch profiler.