You would have to profile the code and isolate the potential bottleneck.
Since the GPU utilization is low, your data loading might be the bottleneck.
This post 115 explains potential workarounds and best practices for this issue.