Solved by ptrblck in post #4 
              
              
                The GPU is already used for the model forward and badckward passes. 
However, since your model is really small, the CPU workload might be the bottleneck and would thus cause a low GPU utilization. 
E.g. using some imaginary numbers: 

GPU forward + backward pass takes 1s
data loading and processing …