In my experience, theVolatile GPU-Util of nvidia-smi should always be 100% or so . Maybe your hdd is too slow. Or maybe your CPU is not good enough to fetch the data.
P.S.  use watch --color -n1 nvidia-smi to see GPU-util