Training speed depends on many factors including the input loading time (due to number of threads, RAM, CPU speed etc), Batch size used etc.
With the nvidia-smi info,I could see that the GPU is under-utilized. Atleast memory-wise.
First step would be to increase the batch size and see if the speed is sufficient enough.