Do you see and peaks in the GPU usage?
Also, could you just pass random data to your GPU and see, if the utilization increases?
You might also want to time your data loading as shown in the ImageNet example 27 to see, if you have a bottleneck there.