Thanks! I will take a look at pyprof. The setup on GTX is very different, it is a Linux cluster  with 4 GPUs in a node. But performance on my local RTX is x3,500 slower than on the cluster.
On RTX I can see that GPU RAM is being used and CUDA cores are heavily used, but the overall GPU usage is around 7%.
I tried to experiment with reducing batch size and number of workers, but it does not seem to make a difference.