If you GPU is not fully utilized, your training might suffer from bottnecks such as data loading or small workloads on the GPU.
In the former case, you could try to use synthetic data and check the GPU utilization again.
If you are dealing with a small model and small batches, the actual computation on the GPU might be too small so that the overhead from the data transfer to the device as well as the kernel launches might be visible.